TestEngineering/Performance/Raptor

Raptor

Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.

Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.

Note: Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI (Bug 1460742).

Running Locally on Firefox

To run Raptor locally just build Firefox and then run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name>

For example to run the raptor tp6 pageload test locally just use:

    mozilla-central$ ./mach raptor-test --test raptor-tp6

Running Locally on Google Chrome

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example to run the raptor-speedometer benchmark on Google Chrome use:

    mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Running Raptor on Try

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

    mozilla-central$ $ ./mach try fuzzy --full

Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.

Raptor Tests

Currently the following Raptor tests are available.

raptor-gdocs

• contact: :rwood, :jmaher
• type: pageload
• browsers: Firefox
• measuring: time-to-first-non-blank-paint
• reporting: Median of 25 pageload values in MS

The raptor-gdocs test loads three pages (a google document, a google sheets spreadsheet, and a google shares presentation) 25x each and measures their load performance (fnbpaint). To test all of the pages, run 'raptor-gdocs'. To only test one of the individual pages, run the corresponding raptor test ('raptor-google-docs-firefox', 'raptor-google-sheets-firefox', or 'raptor-google-slides-firefox').

In each of the test page (doc/slides/sheets) recordings, there is a [hero element] located as part of the blue 'Sign In' button on the top right.

Results are the pageload median time in MS.

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

raptor-motionmark

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: benchmark measuring the time to animate complex scenes
• summarization:
  • subtest: FPS from the subtest, each subtest is run for 15 seconds, repeat this 5 times and report the median value
  • suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)

Note: There are two tests for motionmark, the test names for the Raptor command line are 'raptor-motionmark-animometer and 'raptor-motionmark-htmlsuite'.

raptor-speedometer

• contact: :selena
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: responsiveness of web applications
• reporting: runs/minute score
• data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
• summarization:
  • subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
  • score: geometric mean of the 16 sums

This is the Speedometer javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.

raptor-stylebench

• contact: :emilio
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: speed of dynamic style recalculation
• reporting: runs/minute score

raptor-sunspider

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• TODO

raptor-tp6

• contact: :rwood, :jmaher
• type: pageload
• browsers: Firefox
• measuring: time-to-first-non-blank-paint
• reporting: Median of 25 pageload values in MS

The raptor-tp6 test loads four pages (amazon, facebook, google, youtube) 25x each and measures their load performance (fnbpaint). To test all of the pages, run 'raptor-tp6'. To only test one of the individual pages, run the corresponding raptor test ('raptor-tp6-amazon-firefox', 'raptor-tp6-google-firefox', 'raptor-tp6-facebook-firefox', or 'raptor-tp6-youtube-firefox').

Results are the pageload time in MS.

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

The page recordings contain [hero elements], located a follows:

raptor-tp6-amazon: string description element for first laptop in search results

raptor-tp6-facebook: on the Facebook 'Home' icon

raptor-tp6-google: bigger photo of Obama in search results towards top right

raptor-tp6-youtube: YouTube logo on the top left

raptor-webaudio

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• TODO