Auto-tools/Projects/SpeedTests: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(27 intermediate revisions by 2 users not shown)
Line 1: Line 1:
See results at [http://brasstacks.mozilla.com/speedtests.html].
== Status ==
 
'''Disabled'''.  This was an interesting project, but the results were not widely used, and periodic breakages due to its mostly-but-not-entirely-automated nature resulted in time spent doing maintenance that had no clear benefit.  If any interest in this project rises again, we can resurrect the system and hopefully make improvements to ease maintenance.
 
However, SpeedTests lives on in a new project, a fork designed for measuring HTML game performance: https://github.com/Mozilla-Games/speedtests
 
== Results ==
 
Results were hosted at http://speedtests.mozilla.org/.


== Description ==
== Description ==


SpeedTests is an automated suite of browser tests in which all tests are executed on a series of browsers.  Tests can be freeform but are expected to a) report results and b) have a definite termination point, for which purpose a small set of JavaScript functions are provided.  Likely candidates are the IE 9 Test Drive performance tests, suitably modified.
SpeedTests is a ''semi''automated system that runs a suite of JS-based tests on a series of browsers.  Tests are generally free form but are expected to (a) report results and (b) have a definite termination point, for which purpose a small set of JavaScript functions are provided.  Tests include suitable modified versions of the IE Test Drive tests, [[Kraken]], V8, and test262.


== Architecture ==
== Architecture ==


Conceptually the system can be split into three parts, the browser controller, the report and test server, and the tests themselves.  However they are best understood by looking at the whole process rather than the individual parts.
Conceptually the system can be split into four parts: the tests, the browser controller, the report server, and test server (the latter two may be combined).  However they are best understood by looking at the whole process rather than the individual parts.


The browser controller has a list of browsers likely to be installed on the target platform.  It iterates through the list, executing the browser if it is found, pointing it to the server path /nexttest/, which replies with a redirect to the first test (in string order).  It also has a simple HTTP server running on a special port, which awaits a callback from the browser when the last test has executed.
Upon starting, the browser controller first fetches a list of tests from the server (in case any have been added or removed since last run--this means clients do not have to be updated when tests change).  It has a list of browsers likely to be installed on the target platform, and iterates through the existing browsers, running each one with each test, separately.  The controller loads a fresh profile before executing each test, and shuts down the browser after each test finishes, to eliminate interactions between tests.  It has a simple HTTP server running on a special port, which proxies the tests from the test server, to get around cross-domain restrictions, and which collects the results from the browser after each test finishes.


The tests use a common set of JavaScript functions to report results and to load the next test.  The server collects results and serves tests.  When the test is finished, the browser sends the results via XHR and then loads the URL /nexttest/<current test name>, eg. /nexttest/PsychedelicBrowsing/, which redirects to the test that, in string order, follows the current test.  If the current test was the last test, the server redirects the browser to the callback port on the localhost.
The tests use a common set of JavaScript functions to collect and report results.  The browser controller collects results from each browser and forwards them onto the results server (the latter can be disabled for local testing).  When the last browser has executed the last test, the controller shuts down its HTTP server, prints a report to stdout, and exits.


Upon receiving a GET request on the callback port, the browser controller terminates the current browser and launches the next one in the list.  When the last browser has executed the tests, the controller shuts down its HTTP server and exits.
The test server can be the same server as the results server, or it can be separate.  In the current setup, they are separated, with the test server running on the internal network, since we can't distribute the modified tests.  It is the test server that also serves up the JS library for reporting results.
 
SpeedTests also supports [https://github.com/markrcote/jwt JSON web tokens] for signing results, in case results must be served over the Internet (for example, the client and the results server being on separate internal networks).
 
The location of the test and result servers is configured in an ini file on the client.


== Implementation ==
== Implementation ==
Line 21: Line 33:
The server is a Python application using the web.py framework along with a MySQL database to store results.
The server is a Python application using the web.py framework along with a MySQL database to store results.


Tests are HTML, JavaScript, and associated files.  The server also provides a file containing several JavaScript functions to be used to report results and load subsequent tests.
Tests are HTML, JavaScript, and associated files.  The server also provides a file containing several JavaScript functions to be used to collect and report results.


Source code is in hg: http://hg.mozilla.org/automation/speedtests/
Source code is in hg: http://hg.mozilla.org/automation/speedtests/
Line 27: Line 39:
Unfortunately, I think legal restrictions prevent us from distributing the Microsoft-originated tests themselves.
Unfortunately, I think legal restrictions prevent us from distributing the Microsoft-originated tests themselves.


== Issues ==
SpeedTests are semiautomated in that nothing is provided to automatically apply OS and browser updates nor disable said updates while a test is in progress.  At the moment, most network access is restricted via DNS to prevent updates, and an admin manually performs the updates every few weeks.  The exception here is Firefox Nightly, which is automatically updated on every run.
 
== Client Configuration ==
 
First, you need a speedtests.conf file alongside speedtests.py. You will need at least a [speedtests] section specifying server locations. For example:
 
[speedtests]
server_url = http://192.168.1.10:8888/api/
test_base_url = http://192.168.1.10:8888/
server_results_url = http://192.168.1.10:8888/api/testresults/
 
In this case, there is a speedtests server running on 192.168.1.10 on port 8888, acting as both a test and a results server. "server_url" refers to the dynamic part of the test server, which is used to get the list of existing tests. "test_base_url" is the root of the html test files (e.g. Kraken, fishtank, etc.). The client sends its results to "server_results_url" after tests finish executing.
 
The speedtests client looks in the usual spot for the browsers, e.g. /usr/bin/firefox on Linux or $PROGRAMFILES\Mozilla\Mozilla Firefox on Windows. You can override these with a [<platform name>] section containing options in the form <browser> = <path>. For example, to override the location of Firefox on Linux, add this to speedtests.conf:
 
[linux]
firefox = /opt/firefox/firefox
 
The main configuration task is unfortunately not easy.  All the browsers must have a stored profile that permits its browser to
* open a new window (all tests are run in a separate window to control window size)
* AJAX calls, to send the results to localhost
* load a page from localhost
 
Some browsers are more particular than others about what is allowed out of the box.
 
The profiles should also have empty caches or a clear-cache-on-exit setting.  This ensures that any changes to the tests on the server will be picked up the next time the clients are run.
 
I have been collecting some stock profiles, but unfortunately they are not always compatible between releases.  The most reliable way to set up the stored profiles is through the "archive" and "load" commands, followed by testing in testmode (see below).
 
Finally, in order to prevent updates to browsers that might affect performance while tests are being run, it is recommended that the network be partially disabled, with access only to the test server.  This can be done by disabling DNS and putting the server IP into the hosts file.  A standalone script, nw.py, has been provided to enable and disable the DNS on Windows machines (must be run as administrator).
 
== Usage ==
 
The client can be started by just running "python speedtests.py".  If you don't want to run all the browsers, you can append a list of desired browsers, e.g. "python speedtests.py nightly 'internet explorer' chrome".  You can also use the "-t" option to provide a test path.  Note that in this case speedtests.py does not fetch the test list from the server, so you have to provide the full (relative) path, e.g. -t test262/default.html.
 
Adding "--noresults" will prevent the client from sending the results back to the test server.  Results will still be output to the terminal.
 
It can also be run in test mode by appending the "--testmode" option.  This causes the server to return simple test pages that do not run any tests but exercise the framework itself, namely, ensuring that the browser can deal with popups, accessing localhost, etc.
 
There are also two commands to help you set up and test the stored profiles:
* archive <browser>: Store the current profile for <browser>
* load <browser>: Starts <browser> with the currently stored profile
 
== Client Maintenance ==
 
As mentioned above, we limit network access on the test machine in order to prevent browser and system updates, which might throw off results if a test is running at the same time.  Unfortunately this requires periodic manual maintenance in order to install desired updates (e.g. to test against recent browser releases), but it should be minimal. It should be possible to do this sort of thing by configuring the host appropriately (using enterprise-level settings/software), but I haven't looked into that yet.
 
Network access is limited by setting the DNS server to localhost, i.e. an invalid host.  Thus all DNS lookups, except those in the hosts file, will fail, preventing the majority of Internet access.  The helper program <tt>nw.py</tt>, distributed with the SpeedTests client, can be used to disable and re-enable DNS.
 
This is the process for updating a SpeedTest client:
 
# Open MozPrompt (on the desktop) as administrator.
# Go to the speedtests client directory (generally /Users/mozilla/speedtests/client).
# Run <tt>python nw.py enable</tt> to enable full network access.
# Run Firefox, Chrome, and Opera, checking each for updates.
# Run the Apple Updater to look for Safari updates (don't install other software like Quicktime!).
# Run Windows Update.
# Run Windows Update again to see if there are subsequent updates.
# Run <tt>python nw.py disable</tt> to disable full network access.
# Reboot.


As with any automation, there are finnicky things needing to be worked out (or around):


* Window size.  For a proper comparison, we want all browsers to run in the same-sized window.  However, not all OSs appear to provide a way to dictate window size when launching an application (testing so far has been on OS X, for which I have not been able to find a suitable solution).
Note that it is ''unnecessary'' to update Nightly, since the client automatically downloads the latest version.
** RESOLVED A start page was added to open the tests themselves in a second window of a set size (currently 1024x768).  There are very slight differences in how browsers interpret sizes specified in window.open() (e.g. window.innerHeight and .innerWidth can vary) but not enough to make much of a difference.


* Security preferences.  Some browsers, notably Opera, don't like pages that try to redirect to localhost.  Some prompt the user to restore the previous session upon launch.  This requires some sort of initial setup; hopefully a saved profile will be sufficient.  Although, Opera sometimes forgets security settings and reverts to user prompts--not useful in an automated environment.
=== Setting up browser profiles ===
** RESOLVED Using a stock profile to overwrite the current profile each time the tests are launched appears to have overcome these problems.


* Frozen tests.  It's entirely possible that a test won't finish, either because of a problem in the test or wonky browser behaviour.  The controller could kill the browser in this case, but the controller doesn't know how long the current test has been running (or even which test it's executing).  Furthermore, we'd probably lose the rest of the test results.<br><br>A possible solution would be to move the test-loading logic to the controller instead of being done in the test itself.  However this prevents a casual user from running the test suite independent of a controller (by just going to http://.../nexttest/).  It also requires finer control over the browser.
This is the trickiest part.  The framework needs relatively clean profiles with nothing cached, with popups enabled for the test server (since tests are opened in a separate window to control the size), and with access to localhost permitted (only an issue with Opera, I believe).  There are also other random things that need to be dismissed, like add-on compatibility dialogs.
** RESOLVED We give the controller a total test time of 10 minutes, after which it kills the browser and moves on.  This obviously can't be used when going straight to the tests URL without using the controller, but data gained the latter way is much harder to use anyway.


* Browser and system updates.  Some browsers check for, and, either silently or with a user prompt, upgrade the browser.  Also Windows updates might be downloaded and/or automatically installed.  This is not desirable since it can throw off results if a test is running at the same time.
When setting up a new client, I first start a test run with --testmode.  This will verify most of the settings without having to sit through all the tests.  If I see a browser not being able to load a test, I stop the controller, set all the correct permissions, clear the cache, and run <tt>python speedtests.py archive &lt;browser name&gt;</tt>, where <tt>&lt;browser name&gt;</tt> is one of <tt>firefox</tt>, <tt>nightly</tt>, <tt>safari</tt>, <tt>"internet explorer"</tt>, or <tt>chrome</tt>.  To retest this browser, I start the tests again in testmode, providing the browser name. However, this won't quite work in Linux, since killing the speedtests client will kill the browser subprocess as well... So this method will have to be refined.
** RESOLVED We limit network access on the machine.  We do this by disabling DNS (by providing an invalid DNS server, e.g. 127.0.0.1) and adding the test server IP address to the hosts file. Unfortunately this requires periodic manual maintenance in order to install desired updates (e.g. to test against recent browser releases), but it should be minimal.


* Sudden drops in performance.  After a few days of running well, several browsers suddenly reported extremely different results, generally much lower.  There were also service interruptions.  These problems continued unabated.
If something funny is still happening, you can run with --noresults, which will execute the full tests but not report results back to the server, so you don't have to worry about affecting the results with VNC open (VNC is *supposed* to not interfere with framerates, but I have my doubts...).
** RESOLVED (I THINK?) After a configuration change, the machine was no longer automatically logging in.  This caused some runs not to occur at all, and some ran through a virtual remote-desktop terminal.  Also, remote desktop can apparently do strange things; there is some evidence that the nightlies stopped working properly after they were launched from remote desktop.  VNC seems to have restored the correct behaviour. VNC should be the only form of access to the machines to eliminate strange problems like this.


== Major Tasks Remaining ==
== Major Tasks Remaining ==


* Set up a second Windows box with different hardware. (8-16 h, maybe more, depending on problems encountered)
* Improve packaging and setup, particularly for dev environments.
** Modify reports slightly to account for different machines running the same platform.  Differentiating by IP or by MAC address may be sufficient.


* Set up a Mac box. (8-16 h)
* Support for mobile browsers (probably via AutoPhone).


* Set up a Linux box. (8-16 h)
* Set up a Mac box.


* Various tweaks to reports as requested by customers (variable, say 16 h total).
* Set up a Linux box.

Latest revision as of 17:43, 30 October 2013

Status

Disabled. This was an interesting project, but the results were not widely used, and periodic breakages due to its mostly-but-not-entirely-automated nature resulted in time spent doing maintenance that had no clear benefit. If any interest in this project rises again, we can resurrect the system and hopefully make improvements to ease maintenance.

However, SpeedTests lives on in a new project, a fork designed for measuring HTML game performance: https://github.com/Mozilla-Games/speedtests

Results

Results were hosted at http://speedtests.mozilla.org/.

Description

SpeedTests is a semiautomated system that runs a suite of JS-based tests on a series of browsers. Tests are generally free form but are expected to (a) report results and (b) have a definite termination point, for which purpose a small set of JavaScript functions are provided. Tests include suitable modified versions of the IE Test Drive tests, Kraken, V8, and test262.

Architecture

Conceptually the system can be split into four parts: the tests, the browser controller, the report server, and test server (the latter two may be combined). However they are best understood by looking at the whole process rather than the individual parts.

Upon starting, the browser controller first fetches a list of tests from the server (in case any have been added or removed since last run--this means clients do not have to be updated when tests change). It has a list of browsers likely to be installed on the target platform, and iterates through the existing browsers, running each one with each test, separately. The controller loads a fresh profile before executing each test, and shuts down the browser after each test finishes, to eliminate interactions between tests. It has a simple HTTP server running on a special port, which proxies the tests from the test server, to get around cross-domain restrictions, and which collects the results from the browser after each test finishes.

The tests use a common set of JavaScript functions to collect and report results. The browser controller collects results from each browser and forwards them onto the results server (the latter can be disabled for local testing). When the last browser has executed the last test, the controller shuts down its HTTP server, prints a report to stdout, and exits.

The test server can be the same server as the results server, or it can be separate. In the current setup, they are separated, with the test server running on the internal network, since we can't distribute the modified tests. It is the test server that also serves up the JS library for reporting results.

SpeedTests also supports JSON web tokens for signing results, in case results must be served over the Internet (for example, the client and the results server being on separate internal networks).

The location of the test and result servers is configured in an ini file on the client.

Implementation

The browser controller is a single Python file, requiring an installation of Python 2.6 or higher. We may build a standalone Windows executable with py2exe for convenience.

The server is a Python application using the web.py framework along with a MySQL database to store results.

Tests are HTML, JavaScript, and associated files. The server also provides a file containing several JavaScript functions to be used to collect and report results.

Source code is in hg: http://hg.mozilla.org/automation/speedtests/

Unfortunately, I think legal restrictions prevent us from distributing the Microsoft-originated tests themselves.

SpeedTests are semiautomated in that nothing is provided to automatically apply OS and browser updates nor disable said updates while a test is in progress. At the moment, most network access is restricted via DNS to prevent updates, and an admin manually performs the updates every few weeks. The exception here is Firefox Nightly, which is automatically updated on every run.

Client Configuration

First, you need a speedtests.conf file alongside speedtests.py. You will need at least a [speedtests] section specifying server locations. For example: 

[speedtests]
server_url = http://192.168.1.10:8888/api/
test_base_url = http://192.168.1.10:8888/
server_results_url = http://192.168.1.10:8888/api/testresults/

In this case, there is a speedtests server running on 192.168.1.10 on port 8888, acting as both a test and a results server. "server_url" refers to the dynamic part of the test server, which is used to get the list of existing tests. "test_base_url" is the root of the html test files (e.g. Kraken, fishtank, etc.). The client sends its results to "server_results_url" after tests finish executing.

The speedtests client looks in the usual spot for the browsers, e.g. /usr/bin/firefox on Linux or $PROGRAMFILES\Mozilla\Mozilla Firefox on Windows. You can override these with a [<platform name>] section containing options in the form <browser> = <path>. For example, to override the location of Firefox on Linux, add this to speedtests.conf: 

[linux]
firefox = /opt/firefox/firefox

The main configuration task is unfortunately not easy. All the browsers must have a stored profile that permits its browser to

  • open a new window (all tests are run in a separate window to control window size)
  • AJAX calls, to send the results to localhost
  • load a page from localhost

Some browsers are more particular than others about what is allowed out of the box.

The profiles should also have empty caches or a clear-cache-on-exit setting. This ensures that any changes to the tests on the server will be picked up the next time the clients are run.

I have been collecting some stock profiles, but unfortunately they are not always compatible between releases. The most reliable way to set up the stored profiles is through the "archive" and "load" commands, followed by testing in testmode (see below).

Finally, in order to prevent updates to browsers that might affect performance while tests are being run, it is recommended that the network be partially disabled, with access only to the test server. This can be done by disabling DNS and putting the server IP into the hosts file. A standalone script, nw.py, has been provided to enable and disable the DNS on Windows machines (must be run as administrator).

Usage

The client can be started by just running "python speedtests.py". If you don't want to run all the browsers, you can append a list of desired browsers, e.g. "python speedtests.py nightly 'internet explorer' chrome". You can also use the "-t" option to provide a test path. Note that in this case speedtests.py does not fetch the test list from the server, so you have to provide the full (relative) path, e.g. -t test262/default.html.

Adding "--noresults" will prevent the client from sending the results back to the test server. Results will still be output to the terminal.

It can also be run in test mode by appending the "--testmode" option. This causes the server to return simple test pages that do not run any tests but exercise the framework itself, namely, ensuring that the browser can deal with popups, accessing localhost, etc.

There are also two commands to help you set up and test the stored profiles:

  • archive <browser>: Store the current profile for <browser>
  • load <browser>: Starts <browser> with the currently stored profile

Client Maintenance

As mentioned above, we limit network access on the test machine in order to prevent browser and system updates, which might throw off results if a test is running at the same time. Unfortunately this requires periodic manual maintenance in order to install desired updates (e.g. to test against recent browser releases), but it should be minimal. It should be possible to do this sort of thing by configuring the host appropriately (using enterprise-level settings/software), but I haven't looked into that yet.

Network access is limited by setting the DNS server to localhost, i.e. an invalid host. Thus all DNS lookups, except those in the hosts file, will fail, preventing the majority of Internet access. The helper program nw.py, distributed with the SpeedTests client, can be used to disable and re-enable DNS.

This is the process for updating a SpeedTest client:

  1. Open MozPrompt (on the desktop) as administrator.
  2. Go to the speedtests client directory (generally /Users/mozilla/speedtests/client).
  3. Run python nw.py enable to enable full network access.
  4. Run Firefox, Chrome, and Opera, checking each for updates.
  5. Run the Apple Updater to look for Safari updates (don't install other software like Quicktime!).
  6. Run Windows Update.
  7. Run Windows Update again to see if there are subsequent updates.
  8. Run python nw.py disable to disable full network access.
  9. Reboot.


Note that it is unnecessary to update Nightly, since the client automatically downloads the latest version.

Setting up browser profiles

This is the trickiest part. The framework needs relatively clean profiles with nothing cached, with popups enabled for the test server (since tests are opened in a separate window to control the size), and with access to localhost permitted (only an issue with Opera, I believe). There are also other random things that need to be dismissed, like add-on compatibility dialogs.

When setting up a new client, I first start a test run with --testmode. This will verify most of the settings without having to sit through all the tests. If I see a browser not being able to load a test, I stop the controller, set all the correct permissions, clear the cache, and run python speedtests.py archive <browser name>, where <browser name> is one of firefox, nightly, safari, "internet explorer", or chrome. To retest this browser, I start the tests again in testmode, providing the browser name. However, this won't quite work in Linux, since killing the speedtests client will kill the browser subprocess as well... So this method will have to be refined.

If something funny is still happening, you can run with --noresults, which will execute the full tests but not report results back to the server, so you don't have to worry about affecting the results with VNC open (VNC is *supposed* to not interfere with framerates, but I have my doubts...).

Major Tasks Remaining

  • Improve packaging and setup, particularly for dev environments.
  • Support for mobile browsers (probably via AutoPhone).
  • Set up a Mac box.
  • Set up a Linux box.
Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Projects/SpeedTests&oldid=742197"