Overview

At present, the different web projects within Mozilla have a variable approach to testing. The best example is probably AMO, which has a reasonably large (but not comprehensive) set of unit tests, written with SimpleTest.

The goal is to implement a standard testing framework that can be used on different webdev projects, integrating unit, UI, and load tests into a system that can be either executed from the command line or viewed from a dashboard.

Project Definition

• Analyze and select a set of testing tools appropriate for Mozilla's websites. These should cover:
  • Unit testing: At present we have some SimpleTest tests. We should look at using PHPUnit, both for its better PHP5 handling and better community support, and other options.
  • UI testing: At present we have no automated UI tests. We would like to have tests implemented using a tool or tools such as Selenium or FireWatir. We need to think about how we handle non-Firefox UI testing as well. In addition, it would be good to incorporate JavaScript Unit testing, with e.g. JSUnit.
  • Load testing: We have various internal tools available for this, such as Grinder. There may also be other options.

• Implement a pluggable framework that allows the test types above to be integrated into a single environment.

• Enable execution of tests either from the command line or from a browser.

• Implement a dashboard that allows all tests to be executed and the results viewed from a browser. Ideally we should be able to track test coverage and pass rate over time.

• Implement a set of tests for at least one Mozilla web project, most likely AMO. Achieving complete code coverage may not be feasible in the time allowed, but there should be representative tests of each type.

Other requirements

• The test system should work for PHP web apps, and be extensible to cover other types of application. This most likely means that other unit testing frameworks should be pluggable into the system. Bear this required extensibility in mind while designing and implementing the system.

• The system should be configurable as much as possible, e.g. urls to be tested, locations of tests, types of tests, plugins, should all be configurable.

• Documentation of how the system works, how to write tests, and how to extend the framework.

Ideas

Some other ideas:

• Incorporate yslow: http://www.yahooapis.com/yslow/

• As well as load testing (gives request/second) consider the time per request, the number of queries executed per request, any other benchmarks.

• Consider crawling for dead links

• Reporting of PHP errors/warnings/notices

• Run tests automatically over night and emailing key highlights to team members

• Automatic profiling and output integrated via webgrind http://code.google.com/p/webgrind/

• Send updates via IRC.

Resources

• A good way to get started would be to review what we have already: the AMO tests can be accessed by checking out AMO from http://svn.mozilla.org/addons/trunk. Tests are located in site/app/tests.
  • Documentation: http://wiki.mozilla.org/Update:Remora_Testing

• Some testing tools:
  • PHPUnit: http://www.phpunit.de/
  • Selenium: http://selenium.openqa.org/
  • Grinder: http://grinder.sourceforge.net/. You may also find http://wiki.mozilla.org/Bouncer:GrinderLoadTesting helpful.
  • JSUnit: http://www.jsunit.net/