TestEngineering/Performance/Raptor

Raptor

Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.

Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.

Note: Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI (Bug 1460742).

Running Raptor Locally on Firefox

To run Raptor locally just build Firefox and then run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name>

For example to run the raptor tp6 pageload test locally just use:

    mozilla-central$ ./mach raptor-test --test raptor-tp6

Running Raptor Locally on Google Chrome

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example to run the raptor-speedometer benchmark on Google Chrome use:

    mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Running Raptor on Try

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected on turned on in the Treeherder 'Tiers' menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

    mozilla-central$ $ ./mach try fuzzy --full

Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.