TestEngineering/Performance/Raptor: Difference between revisions

* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DRaptor

=== Page-Load Tests ===

===== Fennec =====

===== Fenix =====

===== GeckoView =====

===== Reference Browser =====

'''NOTE:'''

To see which Raptor performance tests are currently available on all platforms, use the 'print-tests' option, e.g.:

That will output all available tests on each supported app, as well as each subtest available in each suite (i.e. all the pages in a specific page-load tp6* suite).

To run Raptor locally, just build Firefox and then run:

For example, to run the raptor-tp6 pageload test locally, just use:

You can run individual subtests too (i.e. a single page in one of the tp6* suites). For example, to run the amazon page-load test on Firefox:

Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.

Once your Android device is ready, and attached to local USB, from within your local mozilla repo use the following command line to run speedometer:

Note: Speedometer on Android GeckoView is currently running on two devices in production - the Google Pixel 2 and the Moto G5 - therefore it is not guaranteed that it will run successfully on all/other untested android devices. There is an intermittent failure on the Moto G5 where speedometer just stalls ([https://bugzilla.mozilla.org/show_bug.cgi?id=1492222 Bug 1492222]).

To run a Raptor page-load test (i.e. tp6m-1) on the GeckoView Example app, use this command line:

A couple notes about debugging:

To run Raptor locally on Google Chrome, make sure you already have a local version of Google Chrome installed, and then from within your mozilla-repo run:

For example, to run the raptor-speedometer benchmark on Google Chrome use:

Raptor test results will be found locally in <your-repo>/testing/mozharness/build/raptor.json.

You can override the default page-timeout by using the --page-timeout command-line arg. In this example, each test page in tp6-1 will be given two minutes to load during each page-cycle:

If an iteration of a benchmark test is not finishing within the allocated time, increase it by:

==== Page-Cycles ====

You can override the default page-cycles by using the --page-cycles command-line arg. In this example, the test page will only be loaded twice:

==== Running Page-Load Tests on Live Sites ====

To tell Raptor to create Gecko profiles during a performance test, just add the '--gecko-profile' flag to the command line, i.e.:

When the Raptor test is finished, you will be able to find the resulting gecko profiles (ZIP) located locally in:

     mozilla-central/testing/mozharness/build/blobber_upload_dir/

Raptor will automatically launch Firefox and load the latest Gecko profile in [https://perf-html.io perfhtml.io]. To turn this feature off, just set the DISABLE_PROFILE_LAUNCH=1 env var.

All the pages in a test suite an be run by calling the top-level test name, i.e.:

Individual test pages can be ran by calling the subtest, i.e.:

Some of the page recordings contain [[https://wiki.mozilla.org/Performance_sheriffing/Raptor/Mitmproxy#Adding_Hero_Elements hero elements]]. When hero elements are measured, the value is the time until the hero element appears on the page (in MS).

The easiest way to debug the Raptor web extension is to run the Raptor test locally and invoke debug mode, i.e. for Firefox:

Or on Chrome, for example:

Running Raptor with debug mode will:

In order to retrieve the console.log() output from the Raptor runner, do the following:

# During the 30 second Raptor pause which happens right after Firefox has started up, in the ALREADY OPEN current tab, type "about:debugging" for the URL.

# On the debugging page that appears, make sure "Add-ons" is selected on the left (default).

When running Raptor tests on Firefox on Android (i.e. geckoview), to see the console.log() output from the Raptor web extension, do the following:

# Start up a local copy of the Firefox Nightly Desktop browser

# In Firefox Desktop choose "Tools => Web Developer => WebIDE"