The QA team was created on July 11, 2005, two days after the release of Bugzilla 2.18.3, to improve the quality of future releases. Bugzilla 2.20, which was released two months later, became our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA team which found several tens of bugs. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team had less than 10 members) to test the most important features of Bugzilla.
As repeating the same tests manually again and again for each new release quickly became boring, we decided to automate the process as much as possible using Selenium. For Bugzilla 2.20.x, 2.22.x and 3.0.x, we used test installations on landfill, which were running HTML Selenium scripts. Since Bugzilla 3.2 RC1, the QA team uses Perl Selenium scripts, which offer many advantages over their HTML equivalent.
How to contribute?
As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the #qa-bugzilla channel on IRC, or to write to email@example.com telling us that you are interested in contributing. Of course, you can also report bugs you discover to b.m.o directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.
Running Selenium scripts
Prior to Bugzilla 3.2, Selenium tests were all written in HTML and executed from a web browser. As of Bugzilla 3.2 RC1 they have been converted to Perl, allowing more flexibility and control.
A read-only copy of the current Selenium scripts can be downloaded using bzr with:
bzr co bzr://bzr.mozilla.org/bugzilla/qa/4.0
A read/write copy can also be downloaded with:
bzr co bzr+ssh://firstname.lastname@example.org/bugzilla/qa/4.0
where login is your LDAP account. All available releases are listed here.
Note: The HTML scripts available in 3.0 and below won't work as is. They were based on a particular landfill installation which contained specific user accounts, products, components and parameters. Thus, they will not run on a fresh test installation. They have been removed from 3.2.
As of Bugzilla 3.2, a script named config/generate_test_data.pl will automatically populate new installations, allowing the Selenium and WebService tests to be executed.
Note 1: The Bugzilla installation to test must already exist, and must already have its important parameters configured (i.e. urlbase, cookiepath, and mail_delivery_method). It is recommended that mail_delivery_method be set to Test, unless "actual" bug mail is desired.
Note 2: Before executing generate_test_data.pl, make sure the parameters in config/selenium_test.conf are set correctly. This configuration file must match the Bugzilla configuration, especially the urlbase parameter and the path to the browser (Selenium RC 2.0 works with Firefox 5 and IE9). If mail_delivery_method is set to Test, fake user accounts can be defined in the config file.
Once the DB is populated, start the Selenium server and execute the scripts in t/. To start the Selenium server, type:
java -jar /path/to/selenium-server-standalone-2.x.x.jar
The Selenium Server can be downloaded from the Selenium website. To execute the scripts, go into the t/ directory and enter:
prove -v --timer *.t
-v will make the output verbose and --timer will display the time it takes to execute each script. Both options are optional. Note that the Test::WWW::Selenium Perl module must be installed in order for the scripts to execute. It is the interface between Perl and Selenium!
Writing Selenium scripts
The fastest way to write new Selenium scripts is to use the Selenium IDE extension for Firefox (note that Selenium IDE 1.0.12 is the last version to export Selenium scripts to the Perl language! Starting with version 1.1.0, the Perl exporter is no longer available). It records all your actions and converts them into a valid Selenium script. You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.
Each command is of the form $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. All available commands are listed here, and are fully documented.
There are many bugs which need a test. You can pick one of them and write a test, either by improving one of the existing tests, or by writing a completely new one. There are also some other existing scripts which need improvements. If you want to submit a script, please file it here.