Confirm
2,177
edits
Changes
no edit summary
[[Image:Raptor.png|frameless|right]]
Raptor is a performance-testing framework for running browser pageload and browser benchmark tests. The core of Raptor was designed as a browser extension, therefore Raptor is cross-browser compatible and is currently running in production on Firefox Desktop, Firefox Android GeckoView, Fennec, Fenix, Reference Browser, Chromium, and Chrome.
* Contact: Dave Hunt [:davehunt]
Raptor currently supports three test types: 1) page-load performance tests, 2) standard benchmark-performance tests, and 3) "scenario"-based tests, such as power, CPU, and memory-usage measurements on Android (and desktop?).
Locally, raptor Raptor can be invoked with either of the following commands - raptor-test may be deprecated in the futurecommand:
./mach raptor
=== Page-Load Tests ===
* The Firefox android app is started up
* Post-startup browser settle pause of 30 seconds
* The test URL is loaded; measurements taken
* The tab is reloaded 14 more times; measurements taken each time
* The Firefox Android app is started up
* Post-startup browser settle pause of 30 seconds
* The test URL is loaded; measurements taken
* The Android app is shut down
==== Using Live Sites ====
It is possible to use live web pages for the page-load tests instead of using the mitproxy recordings. This option is available when running on Try only; as we don't want to submit data from live pages to Perfherder (since live page content will always be changing).
To run a particular Raptor tp6 page-load test with live sites, open the raptor-tp6*.ini file ([https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests Raptor tests folder]), and for the test default (or under a single page/subtest) just add this attribute:
It is possible to disable alerting for all our performance tests. Open the target test manifest such as the raptor-tp6*.ini file ([https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests Raptor tests folder]), and make sure there are no <tt>alert_on</tt> specifications.
When it's removed there will no longer be a <tt>shouldAlert</tt> field in the output perferhder Perfherder data (you can find the [https://searchfox.org/mozilla-central/source/testing/mozharness/external_tools/performance-artifact-schema.json#68,165 schema here]). As long as <tt>shouldAlert</tt> is not in the data, no alerts will be generated. If you need to also disable code sheriffing for the test, then you need to change the tier of the task to 3.
==== High value tests ====
For a combined-measurement run with distinct Perfherder output for each measurement type, you can do:
./mach raptor-test --test raptor-scn-power-idle-bg-fenix --app fenix --binary org.mozilla.fenix.performancetest --host 10.0.0.16 --power-test --memory-test --cpu-test
Each measurement subtype (power-, memory-, and cpu-usage) will have a corresponding PERFHERDER_DATA blob:
* We set `scenario_time` to '''20 minutes''' (1200000 milliseconds), and `page_timeout` to '22 minutes' (1320000 milliseconds)
** It's crucial that `page_timeout` exceed `scenario_time`; if not, measurement tests will fail/bail early
* We launch the {Fenix, Fennec, GeckoView, Reference Browser} on-Android app
* Post-startup browser settle pause of 30 seconds
* On Fennec only, a new browser tab is created (other Firefox apps use the single/existing tab)
* Raw power-use measurement data is listed in the perfherder-data.json/raptor.json artifacts
In the Perfherder (or Firefox Health) dashboards for these power usage tests, all data points have milli-Ampere-hour units, with a lower value being better.
Proportional power usage is the total power usage of hidden battery sippers that is proportionally "smeared"/distributed across all open applications.
To run on a tethered phone via USB from a macOS host, on:
===== Fenix =====
* pgo builds tend to be -- from my limited empirical evidence -- about 10 - 15 minutes longer to complete than their opt counterparts
==== Perf Dashboards ====
=== Running Locally ===
* Device is in 'superuser' mode
* The geckoview example app is already installed on the device (from ./mach bootstrap, above). Download the geckoview_example.apk from the appropriate [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=android%2Cbuild android build on treeherder], then install it on your device, i.e.:
With that setting, Raptor will not start the playback tool (i.e. Mitmproxy) and will not turn on the corresponding browser proxy, therefore forcing the test page to load live.
=== Running Raptor on Try ===
==== Raptor Hardware in Production ====
The Raptor performance tests run on dedicated hardware (the same hardware that the Talos performance tests use). See the [[https:/TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Misc#Hardware_Profile_of_machines_used_in_automationPlatforms|Talos hardware used in automation wiki pageperformance platforms]] for more details. ==== Running Fennec ESR 68 tests ==== Fennec 68 tests are setup to run on latest fennec esr 68 build. To start a try run on Fennec ESR 68 run: $ ./mach try fuzzy -q="fennec68" --full
=== Profiling Raptor Jobs ===
! Project !! Repository !! Tests results !! Schedule
|-
| Fenix (aka Firefox Preview) || [https://github.com/mozilla-mobile/fenix/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=fenix Treeherder view] || Every 24 hours [https://tools.taskcluster.net/hooks/project-releng/cron-task-mozilla-mobile-fenix%2Fraptor Taskcluster force hook]
|-
| Reference-Browser || [https://github.com/mozilla-mobile/reference-browser/ Github] || [https://treeherder.mozilla.org/#/jobs?repo=reference-browser Treeherder view] || On each push
* https://firefox-source-docs.mozilla.org/tools/lint/coding-style/coding_style_python.html
[https://github.com/psf/black/ black]is the tool used to reformat the Python code.
