TestEngineering/Performance/Raptor/Browsertime

From MozillaWiki
Jump to: navigation, search

Browsertime is a harness for running performance tests, similar to Mozilla's Raptor testing framework. Browsertime is written in Node.js and uses Selenium WebDriver to drive multiple browsers including Chrome, Chrome for Android, Firefox, and (pending the resolution of Bug 1525126 and similar tickets) Firefox for Android and GeckoView-based vehicles.

Source code:

Page-load tests

Running Locally

Prerequisites

⚠️ If ffmpeg is listed as FAIL, you might want to try this:

  • cd ~/.mozbuild
  • cd browsertime
  • cd bin
  • chmod +x ffmpeg ffplay ffprobe

Bugs which might further help are bug 1559168, bug 1559727, and bug 1574964, for starters

  • If you plan on running Browsertime on Android, your Android device must already be set up (see more below in the Android section)

Running on Firefox Desktop

  • ./mach browsertime https://www.sitespeed.io --firefox.binaryPath '/Users/sdonner/moz_src/mozilla-unified/obj-x86_64-apple-darwin18.7.0/dist/Nightly.app/Contents/MacOS/firefox'

Running on Fennec

  • .

Running on Fenix/Firefox Preview

  • ./mach browsertime --android --browser firefox --firefox.android.package org.mozilla.fenix.debug --firefox.android.activity org.mozilla.fenix.IntentReceiverActivity https://www.sitespeed.io

Running on the GeckoView Example app

Running on Reference Browser

  • .

Running on Google Chrome

More Examples

Browsertime docs

Running Browsertime on Try

Include the --browsertime flag to mach try fuzzy. As of bug 1566174 this will arrange for the jobs to include dependencies for Browsertime (Node.js, Browsertime node_modules, geckodriver, chromedriver, etc) and for the Raptor harness to run the given pageload test using Browsertime.

Not all Raptor tasks use Browsertime meaningfully. Right now, only cold pageload tests are supported. See bug 1577905 for warm pageload tests, for example.

For example:

 ./mach try fuzzy --browsertime -q "'test-android-hw-g5-7-0-arm7-api-16/opt-raptor-tp6m-1-geckoview-e10s"

Bugs

Full Query
ID Summary Priority Status
1543247 Add a `mach browsertime` command to make it easy to install and invoke browsertime -- RESOLVED
1561939 [meta] MVP for browsertime on Windows 10 in CI -- NEW
1562870 Generate Conditioned profiles P2 RESOLVED
1563074 Lift Android device performance tuning out of `testing/raptor/raptor.py` P2 RESOLVED
1563090 Add a visualmetrics task type for computing visualmetrics from captured video -- RESOLVED
1564256 Expose browsertime to generic-worker based test tasks -- RESOLVED
1564282 Refactor raptor/raptor.py in preparation for browsertime -- RESOLVED
1565316 Process browsertime results and dump out for perfherder P1 RESOLVED
1566166 Add automation to install browsertime on Win 10 production performance hardware P1 RESOLVED
1566171 Expose browsertime-* paths to Raptor test harness P1 RESOLVED
1566174 Taskcluster config for raptor browsertime win10 page-load test P2 VERIFIED
1568048 Lift mozproxy management out of `Raptor` and into `Perftest` P1 RESOLVED
1572804 Begin to build browsertime support into Raptor framework P1 RESOLVED
1574182 [meta] MVP for browsertime on non-Windows (Android, Linux, macOS) in CI -- NEW
1577905 Support warmload tests under --browsertime P2 NEW
1581580 Add limited set of --browsertime Raptor tests to make it easier to smoke-test changes P1 ASSIGNED

16 Total; 4 Open (25%); 11 Resolved (68.75%); 1 Verified (6.25%);