TestEngineering/Performance/Raptor

Raptor

Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.

Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.

Note: Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI (Bug 1460742).

Running Locally on Firefox

To run Raptor locally just build Firefox and then run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name>

For example to run the raptor tp6 pageload test locally just use:

    mozilla-central$ ./mach raptor-test --test raptor-tp6

Running Locally on Google Chrome

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example to run the raptor-speedometer benchmark on Google Chrome use:

    mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Running Locally on Android Geckoview

<TODO>

Running Raptor on Try

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

    mozilla-central$ $ ./mach try fuzzy

Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.

Raptor Desktop Tests

Currently the following Raptor tests are available to run on the desktop browser(s).

raptor-gdocs

• contact: :rwood, :jmaher
• type: pageload
• browsers: Firefox only (for now)
• measuring: time-to-first-non-blank-paint, and hero element load time
• reporting: For each test page both fnbpaint and time to hero element display are measured. Each pagecycle reports the mean of those two values (in MS). There are 25 pagecycles, with the first pagecycle being dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the 24 values reported for each pagecycle (in MS).

The raptor-gdocs test loads three pages (a google document, a google sheets spreadsheet, and a google shares presentation) 25x each and measures their load performance (fnbpaint, and hero element). To test all of the pages, run 'raptor-gdocs'. To only test one of the individual pages, run the corresponding raptor test ('raptor-google-docs-firefox', 'raptor-google-sheets-firefox', or 'raptor-google-slides-firefox').

In each of the test page (doc/slides/sheets) recordings, there is a [hero element] located as part of the blue 'Sign In' button on the top right.

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

To see the specific google document URLs and other raptor test settings, you can view the raptor-gdocs.ini.

raptor-motionmark

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: benchmark measuring the time to animate complex scenes
• summarization:
  • subtest: FPS from the subtest, each subtest is run for 15 seconds, repeat this 5 times and report the median value
  • suite: we take a geometric mean of all the subtests (9 for animometer, 11 for html suite)

Note: There are two tests for motionmark, the test names for the Raptor command line are 'raptor-motionmark-animometer and 'raptor-motionmark-htmlsuite'.

raptor-speedometer

• contact: :selena
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: responsiveness of web applications
• reporting: runs/minute score
• data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
• summarization:
  • subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
  • score: geometric mean of the 16 sums

This is the Speedometer javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and raptor harness.

raptor-stylebench

• contact: :emilio
• type: benchmark
• browsers: Firefox and Google Chrome
• measuring: speed of dynamic style recalculation
• reporting: runs/minute score

raptor-sunspider

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• TODO

raptor-tp6

• contact: :rwood, :jmaher
• type: pageload
• browsers: Firefox only (for now)
• measuring: time-to-first-non-blank-paint, and hero element load time
• reporting: For each test page both fnbpaint and time to hero element display are measured. Each pagecycle reports the mean of those two values (in MS). There are 25 pagecycles, with the first pagecycle being dropped due to the initial extra loading time/noise. The overall result reported for each test page is the median of the 24 values reported for each pagecycle (in MS).

The raptor-tp6 test loads four pages (amazon, facebook, google, youtube) 25x each and measures their load performance (fnbpaint and hero element display). To test all of the pages, run 'raptor-tp6'. To only test one of the individual pages, run the corresponding raptor test ('raptor-tp6-amazon-firefox', 'raptor-tp6-google-firefox', 'raptor-tp6-facebook-firefox', or 'raptor-tp6-youtube-firefox').

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

The page recordings contain [hero elements], located a follows:

raptor-tp6-amazon: string description element for first laptop in search results

raptor-tp6-facebook: on the Facebook 'Home' icon

raptor-tp6-google: bigger photo of Obama in search results towards top right

raptor-tp6-youtube: YouTube logo on the top left

To see the specific test URLs and other raptor test settings, you can view the raptor-tp6.ini.

raptor-webaudio

• contact: ?
• type: benchmark
• browsers: Firefox and Google Chrome
• TODO

Raptor Android Tests

Currently the following Raptor tests are available to run on android devices.

raptor-speedometer

• contact: :?
• type: benchmark
• devices: android (in production: Google Pixel 2, Moto G5)
• measuring: responsiveness of web applications
• reporting: runs/minute score
• data: there are 16 subtests in Speedometer; each of these are made up of 9 internal benchmarks.
• summarization:
  • subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
  • score: geometric mean of the 16 sums

This is the Speedometer javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and raptor harness. Raptor runs the benchmark using the geckoview example app (found on treeherder in the Android 4.0 API16+ opt build task artifacts).