TestEngineering/Performance/Raptor: Difference between revisions

Revision as of 19:58, 24 July 2018

Raptor

Raptor is a new performance testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production (tier 2) on Firefox and Google Chrome.

Raptor supports two types of performance tests: pageload tests, and standard benchmark tests. Pageload tests basically involve loading a specific web page and measuring the load performance (i.e. time-to-first-non-blank-paint). Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated into Raptor to run per-commit in our production CI.

For pageload tests, instead of using live web pages for performance testing, Raptor uses a tool called [Mitmproxy]. Mitmproxy allows us to record and playback test pages via a local Firefox proxy. The Mitmproxy recordings are stored on tooltool and are automatically downloaded by Raptor when they are required for a test.

Note: Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI (Bug 1460742).

Running Locally on Firefox

To run Raptor locally just build Firefox and then run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name>

For example to run the raptor tp6 pageload test locally just use:

    mozilla-central$ ./mach raptor-test --test raptor-tp6

Running Locally on Google Chrome

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

    mozilla-central$ ./mach raptor-test --test <raptor-test-name> --app=chrome --binary="<path to google chrome binary>"

For example to run the raptor-speedometer benchmark on Google Chrome use:

    mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Running Raptor on Try

Raptor tests can be run on try on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).

Note: Raptor is currently 'tier 2' on Treeherder, which means to see the Raptor test jobs you need to ensure 'tier 2' is selected / turned on in the Treeherder 'Tiers' menu.

The easiest way to run Raptor tests on try is to use mach try fuzzy:

    mozilla-central$ $ ./mach try fuzzy --full

Then type 'raptor' and select which Raptor tests (and on what platforms) you wish to run.

Raptor Tests

Currently the following Raptor tests are available.

raptor-tp6

Type: Pageload

Browser: Firefox only

The raptor-tp6 test loads four pages (amazon, facebook, google, youtube) 25x each and measures their load performance (fnbpaint). To test all of the pages, run 'raptor-tp6'. To only test one of the individual pages, run the corresponding raptor test ('raptor-firefox-tp6-amazon', 'raptor-firefox-tp6-google', 'raptor-firefox-tp6-facebook', or 'raptor-firefox-tp6-youtube').

Results are the pageload time in MS.

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

raptor-gdocs

Type: Pageload

Browser: Firefox only

The raptor-gdocs test loads three pages (a google document, a google sheets spreadsheet, and a google shares presentation) 25x each and measures their load performance (fnbpaint). To test all of the pages, run 'raptor-gdocs'. To only test one of the individual pages, run the corresponding raptor test ('raptor-firefox-google-docs', 'raptor-firefox-google-sheets', or 'raptor-firefox-google-slides').

Results are the pageload time in MS.

The pages are played back from [Mitmproxy] recordings. If you need the HTML page source (outside of the Mitmproxy recording) for debugging, the HTML can be found in our perf-automation github repo.

raptor-speedometer

Type: Benchmark

Browser: Firefox and Google Chrome

Description:

@@ Line 9: / Line 9: @@
 '''Note:''' Currently for Google Chrome, Raptor only supports the standard benchmark tests. In order to support pageload tests on Chrome we need to do a bunch of work to get Mitmproxy working with Chrome automatically in our CI ([https://bugzilla.mozilla.org/show_bug.cgi?id=1460742 Bug 1460742]).
-== Running Raptor Locally on Firefox ==
+=== Running Locally on Firefox ===
 To run Raptor locally just build Firefox and then run:
@@ Line 19: / Line 19: @@
       mozilla-central$ ./mach raptor-test --test raptor-tp6
-== Running Raptor Locally on Google Chrome ==
+=== Running Locally on Google Chrome ===
 To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:
@@ Line 29: / Line 29: @@
       mozilla-central$ ./mach raptor-test --test raptor-speedometer --app=chrome --binary="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome
-== Running Raptor on Try ==
+=== Running Raptor on Try ===
 Raptor tests can be run on [https://treeherder.mozilla.org/#/jobs?repo=try try] on both Firefox and Google Chrome. (Raptor pageload-type tests are not supported on Google Chrome yet, as mentioned above).