Confirm
2,177
edits
Changes
no edit summary
* Taskcluster runs a Talos job on a hardware machine when one is available - this is bootstrapped by [https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py mozharness]
** [https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py mozharness downloads the build, talos.zip (found in [http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json talos.json]), and creates a virtualenv for running the test.
** mozharness [[TestEngineering/Performance/Talos../Running#How_Talos_is_Run_in_Production|configures the test and runs it]]
** After the test is completed the data is uploaded to [https://treeherder.mozilla.org/perf.html#/graphs Perfherder]
* Treeherder displays a green (all OK) status and has a link to [https://treeherder.mozilla.org/perf.html#/graphs Perfherder]
* 13 pushes later, [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] is ran which compares your push to the previous 12 pushes and next 12 pushes to look for a [[TestEngineering/Performance/Talos../Data#Regressions|regression]]
** If a regression is found, it will be posted on [https://treeherder.mozilla.org/perf.html#/alerts Perfherder Alerts]
* data: we load 2 pages 25 times each, collect 2 sets of 25 data points
* summarization
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 2 subtest results.
* reporting: test time in ms (lower is better)
* data: We load 5 urls 1 time each, and repeat for 25 cycles; collecting 25 sets of 5 data points
* summarization
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] five data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the rest; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the the subtest results.
* reporting: test time in ms (lower is better)
* data: 12 cycles of the entire benchmark, each subtest will have 12 data points (see below)
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 24 subtest results.
* '''Lower is better'''
* unit: ms/frame
* data: each test loads 25 times
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] 5 data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
* data: each test loads 25 times
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] 5 data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
* data: there are 36 reported subtests from DAMP which we load 25 times, resulting in 36 sets of 25 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l356 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 36 subtest results.
To run this locally, you'll need to pull down the [[#page_sets|tp5 page set]] and run it in a local web server. See the [[#tp5|tp5 section]] or contact jmaher for more information.
* data: we load the displaylist_mutate.html page five times, measuring pageload each time, generating 5 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This measures the amount of time it takes to render a page after changing its display list. The page has a large number of display list items (10,000), and mutates one every frame. The goal of the test is to make displaylist construction a bottleneck, rather than painting or other factors, and thus improvements or regressions to displaylist construction will be visible. The test runs in ASAP mode to maximize framerate, and the result is how quickly the test was able to mutate and re-paint 600 items, one during each frame.
* data: Dromaeo has 6 subtests which run internal benchmarks, each benchmark reports about 180 raw data points each
summarization:
* subtest: Dromaeo is a custom benchmark which has a lot of micro tests inside each subtest, because of this we use a custom [[TestEngineering/Performance/Talos../Data#dromaeo|dromaeo filter]] to summarize the subtest. Each micro test produces 5 data points and for each 5 data points we take the mean, leaving 36 data points to represent the subtest (assuming 180 points). These 36 micro test means, are then run through a geometric_mean to produce a single number for the dromaeo subtest. [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l527 source: filter.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 6 subtest results.
Each page in the manifest is part of the dromaeo css benchmark. Each page measures the performance of searching the DOM for nodes matching various CSS selectors, using different libraries for the selector implementation (jQuery, Dojo, Mootools, ExtJS, Prototype, and Yahoo UI).
* data: we load the perftest.html page (which generates 4 metrics to track) 25 times, resulting in 4 sets of 25 data points
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l381 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 4 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: 5 cycles of the entire benchmark, each subtest will have 5 data points (see below)
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 4 subtest results.
* '''Lower is better'''
* data: there are 14 subtests in kraken, each subtest is an internal benchmark and generates 10 data points, or 14 sets of 10 data points.
* summarization:
** subtest: For all of the 10 data points, we take the [[TestEngineering/Performance/Talos../Data#mean|mean]] to report a single number.** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 14 subtest results.
This is the [[Kraken]] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
* data: we load the rasterflood_svg.html page ten times, measuring pageload each time, generating 10 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This page animates some complex SVG patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate. The result is how quickly the browser is able to render 600 frames of the animation.
* data: we load the rasterflood_gradient.html page ten times, computing a score each time, generating 10 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This page animates some complex gradient patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate.
* data: we load the session restore index page 10 times to collect 1 set of 10 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 9 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** suite: identical to subtest
* summarization:
** subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
** suite: geometric [[TestEngineering/Performance/Talos../Data#mean|mean]] of the 16 sums
This is the [http://browserbench.org/Speedometer/ Speedometer] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 19 data points
** suite: geometric_mean(subtests)
* data: there are 30 reported subtests from TART which we load 25 times, resulting in 30 sets of 25 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 30 subtest results.
TART is the '''Tab Animation Regression Test'''.
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l449 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 51 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l470 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 51 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the tpaint test window 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
* data: we open a new browser window 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
* '''To run it locally''', you'd need [[#page_sets|tp5n.zip]].
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 4 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 50 subtest results.
{| class="mw-collapsible mw-collapsed wikitable"
* data: we run the tresize test page 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 15 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l205 source: test.py]
** suite: same as subtest result
* summarization:
** subtest: identical to suite
** suite: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 19 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l135 source: test.py]
Starts the browser to display tspaint_test.html with the start time in the url, waits for [[Buildbot/Talos/Tests#Paint_Tests|MozAfterPaint and onLoad]] to fire, then records the end time and calculates the time to startup.
* data: we load 6 pages 25 times each, collecting 6 sets of 25 data points
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] data point, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 6 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 2 svg opacity pages 25 times, resulting in 2 sets of 25 data points
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 2 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 5 svg pages 25 times, resulting in 5 sets of 25 data points
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 5 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 7 svg pages 25 times, resulting in 7 sets of 25 data points
* summarization:
** subtest: [[TestEngineering/Performance/Talos../Data#ignore_first|ignore first]] '''5''' data points, then take the [[TestEngineering/Performance/Talos../Data#median|median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [[TestEngineering/Performance/Talos../Data#geometric_mean|geometric mean]] of the 7 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [[TestEngineering/Performance/Talos../Data#median|median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
* type: Custom GTest micro-benchmarking
* data: Time taken for a GTest function to execute
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[[TestEngineering/Performance/Talos../Sheriffing#Microbench_Policy|Microbench Sheriffing Policy]]] for some notes on how to treat regressions.
== PerfStrip Tests ==
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [[TestEngineering/Performance/Talos../Data#median|median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Servo_StyleSheet_FromUTF8Bytes_Bench parses a sample stylesheet 20 times with Stylo’s CSS parser that is written in Rust. It starts from an in-memory UTF-8 string, so that I/O or UTF-16-to-UTF-8 conversion is not measured.
