TestEngineering/Performance/Raptor/Browsertime

From MozillaWiki
Jump to: navigation, search

Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and Firefox for Android and GeckoView-based vehicles.

Source code:

Running Locally

Prerequisites

Setup

Note that if you are running Raptor-Browsertime then it will get installed automatically and will also update itself once bug 1686064 lands.

  • Run ./mach browsertime --setup
  • To check your setup, run ./mach browsertime --check, which will output something like:
 ffmpeg:   OK 
 convert:  OK 
 compare:  OK
 Pillow:   OK
 SSIM:     OK
  • If ffmpeg is listed as FAIL, you might want to try this:
 cd ~/.mozbuild/browsertime/ffmpeg-4.1.1-macos64-static/bin
 chmod +x ffmpeg ffplay ffprobe}}

Now, try re-running ./mach browsertime --check, with ffmpeg hopefully fixed

  • For other issues, see if ./mach browsertime --setup --clobber fixes it, or deleting the ~/.mozbuild/browsertime folder and running the aforementioned command.
  • If you aren't running visual metrics, then failures in Pillow and SSIM can be ignored.

If convert and compare are also FAIL bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters. If none of the bugs are related to the issue, please file a bug Testing :: Raptor.

  • If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)

Running on Firefox Desktop

Page-load tests

There are two ways to run performance tests through browsertime listed below. Note that `./mach browsertime` should not be used when debugging performance issues with profiles as it does not do symbolication.

  • Raptor-Browsertime (recommended):
./mach raptor --browsertime -t google --no-conditioned-profile
  • Browsertime-"native":
 ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/{userdir}/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'

Benchmark tests

  • Raptor-wrapped:
./mach raptor -t raptor-speedometer --browsertime

Running on Android

Running on Raptor-Browsertime (recommended):

  • Running on Fenix
 ./mach raptor --browsertime -t amazon --no-conditioned-profile --app fenix --binary org.mozilla.fenix
  • Running on Geckoview
 ./mach raptor --browsertime -t amazon --no-conditioned-profile --app geckoview --binary org.mozilla.geckoview_example

Running on vanilla Browsertime:

  • Running on Fenix/Firefox Preview
 ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io
  • Running on the GeckoView Example app
 ./mach browsertime --android --browser firefox https://www.sitespeed.io

Running on Google Chrome

Chrome releases are tied to a specific version of ChromeDriver -- you will need to ensure the two are aligned.

There are two ways of doing this:

1. Download the ChromeDriver that matches the chrome you wish to run from https://chromedriver.chromium.org/ and specify the path:

 ./mach browsertime https://www.sitespeed.io -b chrome --chrome.chromedriverPath <PATH/TO/VERSIONED/CHROMEDRIVER>

2. Upgrade the ChromeDriver version in tools/browsertime/package-lock.json (see https://www.npmjs.com/package/@sitespeed.io/chromedriver for versions). Run npm install.

Launch vanilla Browsertime as follows:

 ./mach browsertime https://www.sitespeed.io -b chrome

Or for Raptor-Browsertime (use chrome for desktop, and chrome-m for mobile):

 ./mach raptor --browsertime -t amazon --app chrome --no-conditioned-profile --browsertime-chromedriver <PATH/TO/CHROMEDRIVER>

More Examples

Browsertime docs

Running Browsertime on Try

You can run all of our browsertime pageload tests through ./mach try fuzzy --full. We use chimera mode in these tests which means that both cold and warm pageload variants are running at the same time.

For example:

 ./mach try fuzzy -q "'g5 'imdb 'geckoview 'vismet '-wr 'shippable"

Retriggering Browsertime Visual Metrics Tasks

You can retrigger Browsertime tasks just like you retrigger any other tasks from Treeherder (using the retrigger buttons, add-new-jobs, retrigger-multiple, etc.).

When you retrigger the Browsertime test task, it will trigger a new vismet task as well. If you retrigger a Browsertime vismet task, then it will cause the test task to be retriggered and a new vismet task will be produced from there. This means that both of these tasks are treated as "one" task when it comes to retriggering them.

There is only one path that still doesn't work for retriggering Browsertime tests and that's happens when you use `--rebuild X` in a try push submission.

For details on how we previously retriggered visual metrics tasks see VisualMetrics (this will stay here for a few months just in case).

Gecko Profiling with Browsertime

To run gecko profiling using Raptor-Browsertime you can add the --gecko-profile flag to any command and you will get profiles from the test (with the profiler page opening in the browser automatically). This method also performs symbolication for you. For example:

 ./mach raptor --browsertime -t amazon --no-conditioned-profile --gecko-profile

Note that vanilla Browsertime does support Gecko Profiling but it does not symbolicate the profiles so it is not recommended to use for debugging performance regressions/improvements.

Upgrading Browsertime In-Tree

To upgrade the browsertime version used in-tree you can run, then commit the changes made to package.json and package-lock.json:

 ./mach browsertime --update-upstream-url <TARBALL-URL>

Here is a sample URL that we can update to: https://github.com/sitespeedio/browsertime/tarball/89771a1d6be54114db190427dbc281582cba3d47

To test the upgrade, run a raptor test locally (with and without visual-metrics --browsertime-visualmetrics if possible) and test it on try with at least one test on desktop and mobile.

Finding the Geckodriver Being Used

If you're looking for the latest geckodriver being used there are two ways:

  • Find the latest one from here: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=geckodriver
  • Alternatively, if you're trying to figure out which geckodriver a given CI task is using, you can click on the browsertime task in treeherder, and then click on the `Task` id in the bottom left of the pop-up interface. Then in the window that opens up, click on `See more` in the task details tab on the left, this will show you the dependent tasks with the latest toolchain-geckodriver being used. There's an Artifacts drop down on the right hand side for the toolchain-geckodriver task that you can find the latest geckodriver in.

If you're trying to test Browsertime with a new geckodriver, you can do either of the following:

  • Request a new geckodriver build in your try run (i.e. through ./mach try fuzzy).
  • Trigger a new geckodriver in a try push, then trigger the browsertime tests which will then use the newly built version in the try push.

Comparing Before/After Browsertime Videos

We have some scripts that can produce side-by-side comparison videos for you of the worst pairing of videos. You can find the script here: https://github.com/gmierz/moz-current-tests#browsertime-side-by-side-video-comparisons

Once the side-by-side comparison is produced, the video on the left is the old/base video, and the video on the right is the new video.

Bugs

Full Query
ID Summary Priority Status
1543247 Add a `mach browsertime` command to make it easy to install and invoke browsertime -- RESOLVED
1561939 [meta] MVP for browsertime on Windows 10 in CI -- RESOLVED
1562870 Generate Conditioned profiles P2 RESOLVED
1563074 Lift Android device performance tuning out of `testing/raptor/raptor.py` P2 RESOLVED
1563090 Add a visualmetrics task type for computing visualmetrics from captured video -- RESOLVED
1564256 Expose browsertime to generic-worker based test tasks -- RESOLVED
1564282 Refactor raptor/raptor.py in preparation for browsertime -- RESOLVED
1565316 Process browsertime results and dump out for perfherder P1 RESOLVED
1566166 Add automation to install browsertime on Win 10 production performance hardware P1 RESOLVED
1566171 Expose browsertime-* paths to Raptor test harness P1 RESOLVED
1566174 Taskcluster config for raptor browsertime win10 page-load test P2 VERIFIED
1568048 Lift mozproxy management out of `Raptor` and into `Perftest` P1 RESOLVED
1572804 Begin to build browsertime support into Raptor framework P1 RESOLVED
1574182 [meta] Browsertime support on android P1 NEW
1577905 Support warmload tests under --browsertime P2 RESOLVED
1581580 Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes P1 RESOLVED
1583284 [meta] Validate --browsertime Raptor tasks against vanilla Raptor harness tasks P1 RESOLVED
1584245 Target Chrome/Chromium browser under --browsertime P2 RESOLVED
1585013 Add gecko profiling support to raptor-browsertime (desktop) P1 RESOLVED
1585046 [meta] Expand the raptor-browsertime unit test coverage P2 RESOLVED
1587500 Make raptor/browsertime return code setting more flexible P3 NEW
1587761 Surface browsertime options in a generic way P3 NEW
1587828 [meta] Refactor raptor-browsertime test INIs and taskcluster configs P2 RESOLVED
1591560 Intermittent Browsertime warm page loads failing on raptor-tp6-facebook P3 RESOLVED
1591875 Break Browsertime's `run_test` method into multiple specific methods P2 RESOLVED
1592333 Improve raptor-browsertime page-load failure messages P1 RESOLVED
1592633 Handling differences in prefs between browsertime and raptor-browsertime P3 NEW
1592746 Raptor-browsertime perfherder_data includes all known measurements P2 RESOLVED
1593109 Add chrome browsertime warm pageload tests P3 NEW
1599424 Convert vismet task output to a perfherder artifact P1 RESOLVED
1601045 Intermittent raptor-main Critical: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-tp6-amazon-firefox, raptor-tp6-facebook-firefox, raptor-tp6-google-firefox, raptor-tp6-youtube-firefox P3 RESOLVED
1601414 Add application name and version in vismet perfherder results P1 RESOLVED
1602577 Update in-tree Browsertime version -- RESOLVED
1602657 Set appropriate value for post-startup-delay in (and only in) CI, for conditioned-profile runs P1 RESOLVED
1602893 Retriggered browsertime vismet tasks only report results from the first task P2 NEW
1603152 For cold page-load set browser cycles to 3 when gecko profiling is enabled P1 RESOLVED
1604638 don't use conditioned profiles in tests P2 RESOLVED
1607851 Upgrade browsertime version used in production to the sitespeedio v8.0.0 tag P1 RESOLVED
1608033 Get rid of custom geckodriver P4 RESOLVED
1608863 Switch to sitespeed.io browsertime P2 RESOLVED
1610389 Investigate Desktop Browsertime issues. P3 NEW
1612042 Enable fission for browsertime pageload tests. P3 RESOLVED
1615347 Change location where temporary results are written to in Browsertime P3 NEW
1616845 Near-perma "OSError: symbolic link privilege not held" in fetch-content P2 RESOLVED
1617239 Disable tier 3 browsertime tests on windows10-64-ref-hw-2017 P1 RESOLVED
1633336 Browsertime visual metrics tests with webrender enabled should run in the general_perf_testing cron P3 NEW
1633519 Raise browsertime error when it fails instead of generic ones P1 RESOLVED
1645187 [meta] Make browsertime run faster P3 NEW
1650133 [meta] Migrate webext desktop pageload tests to browsertime P2 NEW
1650871 Add all desktop pageload tests to browsertime P1 RESOLVED
1664551 [meta] Speed up browsertime P2 RESOLVED
1665911 Node crashes on Windows 7 browsertime tests P2 NEW
1665925 Tumblr pageload tests are perma-failing on browsertime P2 RESOLVED
1668049 Prefix visual-metrics.py output with more information for treeherder P2 RESOLVED

54 Total; 12 Open (22.22%); 41 Resolved (75.93%); 1 Verified (1.85%);