TestEngineering/Performance/Raptor

Raptor

Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.

Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.

Note: Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI (Bug 1460742).

Running Raptor Locally on Firefox

To run Raptor locally just build Firefox and then run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name>

For example to run the raptor tp6 pageload test locally just use:

    mozilla-central$ ./mach raptor-test --test raptor-tp6

Running Raptor Locally on Google Chrome

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example to run the raptor-speedometer benchmark on Google Chrome use:

    mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Running Raptor on Try

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

    mozilla-central$ $ ./mach try fuzzy --full

Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.