Auto-tools/Projects/SpeedTests

Results

See results at http://brasstacks.mozilla.com/speedtests/results.html.

Description

SpeedTests is an automated suite of browser tests in which all tests are executed on a series of browsers. Tests can be freeform but are expected to a) report results and b) have a definite termination point, for which purpose a small set of JavaScript functions are provided. Likely candidates are the IE 9 Test Drive performance tests, suitably modified.

Architecture

Conceptually the system can be split into three parts, the browser controller, the report and test server, and the tests themselves. However they are best understood by looking at the whole process rather than the individual parts.

The browser controller has a list of browsers likely to be installed on the target platform. It iterates through the list, executing the browser if it is found, pointing it to the server path /nexttest/, which replies with a redirect to the first test (in string order). It also has a simple HTTP server running on a special port, which awaits a callback from the browser when the last test has executed.

The tests use a common set of JavaScript functions to report results and to load the next test. The server collects results and serves tests. When the test is finished, the browser sends the results via XHR and then loads the URL /nexttest/<current test name>, eg. /nexttest/PsychedelicBrowsing/, which redirects to the test that, in string order, follows the current test. If the current test was the last test, the server redirects the browser to the callback port on the localhost.

Upon receiving a GET request on the callback port, the browser controller terminates the current browser and launches the next one in the list. When the last browser has executed the tests, the controller shuts down its HTTP server and exits.

Implementation

The browser controller is a single Python file, requiring an installation of Python 2.6 or higher. We may build a standalone Windows executable with py2exe for convenience.

The server is a Python application using the web.py framework along with a MySQL database to store results.

Tests are HTML, JavaScript, and associated files. The server also provides a file containing several JavaScript functions to be used to collect and report results.

Source code is in hg: http://hg.mozilla.org/automation/speedtests/

Unfortunately, I think legal restrictions prevent us from distributing the Microsoft-originated tests themselves.

Configuration

The main configuration task is unfortunately not easy. All the browsers must have a stored profile that permits its browser to

• open a new window (all tests are run in a separate window to control window size)
• AJAX calls, to send the results to the server
• load a page from localhost

Some browsers are more particular than others about what is allowed out of the box.

The profiles should also have empty caches or a clear-cache-on-exit setting. This ensures that any changes to the tests on the server will be picked up the next time the clients are run.

I have been collecting some stock profiles, but unfortunately they are not always compatible between releases. The most reliable way to set up the stored profiles is through the "archive" and "load" commands, followed by testing in testmode (see below).

Finally, in order to prevent updates to browsers that might affect performance while tests are being run, it is recommended that the network be partially disabled, with access only to the test server. This can be done by disabling DNS and putting the server IP into the hosts file. A standalone script, nw.py, has been provided to enable and disable the DNS on Windows machines (must be run as administrator).

Usage

The client can be started by just running "python speedtests.py". If you don't want to run all the browsers, you can append a list of desired browsers, e.g. "python speedtests.py nightly 'internet explorer' chrome".

It can also be run in test mode by appending the "-t" or "--testmode" option. This causes the server to return simple test pages that do not run any tests but exercise the framework itself, namely, chaining the tests and pinging back the local controller.

There are also two commands to help you set up and test the stored profiles:

• archive <browser>: Store the current profile for <browser>
• load <browser>: Starts <browser> with the currently stored profile

Outstanding Issues

• We limit network access on the machine in order to prevent browser and system updates, which might throw off results if a test is running at the same time. Unfortunately this requires periodic manual maintenance in order to install desired updates (e.g. to test against recent browser releases), but it should be minimal.

Major Tasks Remaining

• Set up a Mac box. (8-16 h)

• Set up a Linux box. (8-16 h)

• (to do eventually) The local controller could act as a proxy for the test server. This would allow better logging on the client side in order to diagnose problems and to provide the tester with immediate results.