Bugzilla:QA

QA testing for Bugzilla

QA team

Since July 11, 2005, two days after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. Bugzilla 2.20 has been our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has 9 members) to test the most important features of Bugzilla.

As repeating the same tests manually again and again for each new release quickly became rather boring, we tried to automate the process as much as possible. To help us in this task, the QA team is now working on a Selenium installation on landfill which can either be run from a web browser or from a Perl script directly.

Writing Selenium scripts

Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language.

Scripts run from a web browser

This is the easiest way to use Selenium. After having downloaded and untar'ed the tarball (latest version so far is 0.6.0), scripts simply consist of a table with three columns (command, field, value). The commands available are described here and a template is available in bug 317695 (template.html), as well as a useful Javascript file for most common actions (user-extensions.js).

selenium-IDE(amo, formerly Selenium Recorder) is a Firefox extension which writes scripts for you. It records your actions and converts them into a valid Selenium script. If you decide to install this extension, you don't need to install Selenium separately; everything is included in the XPI package (257 Kb in size; samples and docs are not included though).

Scripts called by a Perl program

The installation and configuration are a bit harder. First download and install Test::WWW::Selenium as well as all its required modules:

requires:

 Alien::Selenium
 Frontier::Client
 Frontier::RPC2
 LWP v5.8
 POE v0.29
 POE::Component::Client::UserAgent
 POE::Component::Server::HTTP
 Test::LongString

build_requires:

 HTTP::Daemon
 Test::More v0.47

When all these modules are installed, you will also need the SeleniumTest.pm module available in bug 317695 which is a simplified version of the one given with the Test-WWW-Selenium tarball. An example of a test script is also provided (012qa.t), which can e.g. be executed with runtests.pl, a Perl script available in the Bugzilla distribution.

As you can see, the syntax is different, but the commands remain mostly the same. Make sure to end all your commands by "_ok" to generate an output on your screen. The format is now $test->command_name_ok('field', 'value', 'description'), where $test is the test object created by SeleniumTest.pm and which will do the interface between your test script and the web installation.

The description is optional, but gives very useful information. A typical output is of the form:

t/012qa....1..7
ok 1 - Load query.cgi
ok 2 - Check title
ok 3 - Enter and check input in the short_desc field
ok 4 - Recheck the input (using a different method)
ok 5 - Submit request (-> buglist.cgi)
ok 6 - Check title
ok 7 - Make sure that no bug is returned (0 bug found)
ok
All tests successful.

What you can read here are the descriptions given for each command of the script. This makes debugging much easier!

How to contribute?

If you are interested in helping us testing Bugzilla, you can join us in #qa-bugzilla on IRC or submit your test scripts in bug 317695 directly.

You can either do manual checks, as we are still doing now till our Selenium installation is complete, or choose one of the two approaches described above to write automated tests. In the later case, and if you don't wan't to install all the required modules mentioned above or cannot get them working, writing scripts called from a web browser is probably what you want to do. We plan to write a conversion tool anyway, allowing us to write scripts in HTML and convert them in Perl and vice versa.

Ja