Changes

Since The QA team has been created on July 11, 2005, [http://www.bugzilla.org/news two days] after the release of Bugzilla 2.18.3, the QA team has been created to improve the quality of future releases. [http://www.bugzilla.org/releases/2.20/ Bugzilla 2.20] has been , which was released two months later, became our most stable version ever released, with many security bugs fixed. This result has been possible partly thanks to the hard work done by the QA teamwhich found several tens of bugs. But all the testing has been done manually, which required both a lot of time and a lot of people (the QA team has had less than 10 members) to test the most important features of Bugzilla.

As repeating the same tests manually again and again for each new release quickly became rather boring, we tried decided to automate the process as much as possible. To help us in this task, the QA team use a using [http://www.openqaseleniumhq.org/selenium/index.html Selenium] installation on . For Bugzilla 2.20.x, 2.22.x and 3.0.x, we used [http://landfill.bugzilla.org/selenium/bugzilla/index.html test installations on landfill] (access restricted, sorry) which can be run from a web browser. That's how Bugzilla 2.20.x, 2.22.x and 3.0.x are being testedusing HTML Selenium scripts. Since Bugzilla 3.2 RC1, the QA team uses [http://bzr.bugzilla.org/selenium/ Perl Selenium scripts written in Perl], which offers offer many advantages over their HTML equivalent.

As Selenium cannot do everything, and because someone has to write these scripts anyway, we are always looking for new testers. If you are interested in helping us making Bugzilla better and more stable, feel free to join us. The best way to start is to join us in the [irc://irc.mozilla.org/qa-bugzilla #qa-bugzilla] channel on IRC, or to write to [mailto:qa@bugzilla.org qa@bugzilla.org] telling us that you are interestedin contributing. Of course, you can also report bugs you discovered discover to [https://bugzilla.mozilla.org/enter_bug.cgi?product=Bugzilla b.m.o] directly. If you want to help with automated tests, you are highly encouraged to read below, and download our existing test scripts.

Since mid-2006, we use the QA team uses a [http://landfill.bugzilla.org/bugzillaqa Testopia installation] (basically, Bugzilla with the Testopia extension) to track testing progress and to let us easily manage remaining tests to do. That's a great tool to help us work efficiently, avoiding testing what has already been (automatically) tested.We still run some tests manually, despite more and more of them are converted to Selenium scripts (and to be honest, the number of tests to run manually now is close to zero). If you don't know how to write Selenium scripts, you can still help us either by writing new testcases, or by running existing ones manually. If you prefer to write Selenium scripts, that's even better!

We still do some == Running Selenium scripts ==Till Bugzilla 3.0.x, Selenium tests manuallywere all written in HTML, despite more which were the executed from a web browser. It has been decided for Bugzilla 3.2 RC1 and more of newer to convert them are converted to Selenium scripts. If you don't know how to write Selenium scriptsPerl, you can still help giving us either by writing new testcases, or by running existing ones manuallymore flexibility and control. If you prefer to We no longer write Selenium test scripts, that's even better!in HTML.

== Writing Selenium scripts ==Scripts executed from a web browser and those called from a Perl script use a different syntax, the first ones being pure HTML, the second ones being written using Perl language. === Scripts running from a web browser ===Till Bugzilla 3.0.x, Selenium tests were all written in HTML. It has been decided for Bugzilla 3.2 RC1 and newer to convert them to Perl, giving us more flexibility and control. We no longer write test scripts in HTML. HTML and/or Perl All our current Selenium scripts can be downloaded using bzr. For instance, to download Perl scripts for Bugzilla 3.4, type:

Do not expect HTML scripts to work as is (for 3.0 and older. These HTML scripts no longer exist in Bugzilla 3.2 and newer)have a script named <em>config/generate_test_data. They were based on a given test installation on landfill having some given user accounts, products, components and parameters, and so they won't run on a pl</em> which populates your fresh test new installation. They are only available automatically so that you can see what we did till Bugzilla 3.0. Note that the 3.0 branch has a t/ directory with test scripts related to run our Selenium and WebService. As these scripts point to the same test installations as above, they also won't run as is tests yourself on a fresh test your own installation.

=== Perl Selenium scripts ===To avoid the problem mentioned above, we highly improved the QA process during the '''Note 1:''' your Bugzilla 3.2 cycle and decided to write a script which would populate your fresh new installation automatically so that you can also run our Selenium and WebService tests. This script is located in <em>config/generate_test_data.pl</em>must already exists, and requires that you first set parameters in <em>config/selenium_test.conf</em> correctly. It must match your local configuration, especially the url to your installation, which must already exist have configured most important parameters (but should contain nothing more than what checksetup.pl createdsuch as the urlbase, i.e. your admin accountcookiepath, the TestProduct and system groups), as well as the path to your browser mail_delivery_method* parameters (if possible, use Firefox 2. In Firefox 3, the EULA license is displayed on startup with a new profile, which prevents Selenium to work). If you * set mail_delivery_method it to 'TestfileTest'ideally, unless you can define fake user accounts in the config file. Once the DB is populated, you can start the Selenium server and run our scripts located in t/really want to get bugmail)).

The fastest way to write Selenium scripts is to use [http'''Note 2://openqa''' Before running <em>generate_test_data.orgpl</selenium-ideem>, make sure that you set parameters in <em>config/ selenium-IDE] ([https://addons.mozillaselenium_test.org/extensionsconf</moreinfoem> correctly.php?application=firefox&category=Newest&numpg=10&id=2079 amo]This configuration file must match your local configuration, especially the url to your installation (urlbase), formerly as well as the path to your browser ([http://seleniumrecorder.mozdevseleniumhq.org/download/ Selenium RecorderRC 1.0.1]). Selenium-IDE is works like a charm with Firefox extension which writes scripts for you3. It records your actions and converts them into a valid Selenium script5). If you decide set mail_delivery_method to install this extension'Test', you don't need to install Selenium separately; everything is included can define fake user accounts in the XPI package (samples and docs are not included though). You can also write Selenium scripts with a text editor, but this is longer and can be pretty painfulconfig file.

In order to Once the DB is populated, you can start the Selenium server and run these Perl our scriptslocated in t/. To start the Selenium server, you first have to install 2 Perl modules, both of can either run <em>config/selenium_server_start.t</em> (which are available on CPAN: requires the [http://search.cpan.org/~lukec/Alien-SeleniumRC/ Alien::SeleniumRC]Perl module to be installed) or you can type: [http:java -jar /path/search.cpan.org/~lukecto/Testselenium-WWW-Selenium/ Test::WWW::Selenium]server.jar

Each command which is exactly what Alien::SeleniumRC is doing. If you didn't install Alien::SeleniumRC, then of the form $selyou must extract <em>selenium-server.jar</em>command_name_okfrom the [http://seleniumhq.org/download/ Selenium RC] ZIP file ('field'generally, 'value', 'description'), where $sel the JAR file provided by SeleniumRC is more recent than the test object created by Testone found in Alien::WWW::Selenium and which will do SeleniumRC). To run the interface between your test script and your Bugzilla installation. The description is optionalscripts, but gives very useful information. A typical output is of go into the formt/ directory and type: prove -v --timer *.t

#perl test_enter_new_bug.t ok 1 - open, /bugzilla/ ok 2 v will make the output verbose and - Enter admin login name ok 3 - Enter admin password ok 4 - Submit credentials ok 5 - open, timer will display the time it takes to run each script. Both options are optional. Note that you must have the [http:/bugzilla/enter_bugsearch.cpan.cgi?product=TestProduct ok 6 org/~lukec/Test-WWW- Display enter_bug.cgi for Selenium/ Test::WWW::Selenium] Perl module installed in order to have the scripts to run as it will be the selected product (bypass classifications)interface between Perl and Selenium! ok 7 - Enter bug summary ok 8 - Enter bug description== Writing Selenium scripts == ok 9 - Submit bug data The fastest way to write new Selenium scripts is to post_buguse the [http://seleniumhq.org/download/ Selenium IDE] extension for Firefox.cgi ok 10 - Bug created 1It records all your actions and converts them into a valid Selenium script.You can also write Selenium scripts manually with a text editor, but this is longer and can be pretty painful.10

What you can read here are the descriptions given for each Each command is of the scriptform $sel->command_name_ok('field', 'value', 'description'), where $sel is the test object created by Test::WWW::Selenium. The description is optional, but may give some useful information. This makes debugging much easier!

Selenium scripts being under development/review can be found in open bugs depending on [http://landfill.bugzilla.org/bugzillaqa/show_bug.cgi?id=3065 bug 3065]. Those shouldn't be considered as "safe" till they are available using via bzr. If you want to submit a script, please file it [http://landfill.bugzilla.org/bugzillaqa/enter_bug.cgi?product=QA%20tests&component=Selenium%20scripts here], which is the Bugzilla installation used by the QA team.