Confirm
2,177
edits
Changes
Fix internal wiki links
=== Startup Tests ===
[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test Startup tests] launch Firefox and measure the time to the onload or paint events. We run this in a series of cycles (default to 20) to generate a full set of data. Tests that currently are startup tests are:
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#ts_paint |ts_paint]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#tpaint |tpaint]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#tresize |tresize]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#sessionrestore.2Fsessionrestore_no_auto_restore.2Fsessionrestore_many_windows |sessionrestore[_no_auto_restore/_many_windows]]]
=== Page Load Tests ===
Tests which are imported benchmarks are:
* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#ares6 |ARES-6]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#JSS.2FDomaeo_Tests |Dromaeo]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#jetstream |JetStream]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#kraken |Kraken]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#motionmark |MotionMark]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#speedometer |speedometer]]* [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#stylebench |stylebench]]
=== Row Major vs. Column Major ===
* data: we load 2 pages 25 times each, collect 2 sets of 25 data points
* summarization
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 2 subtest results.
* reporting: test time in ms (lower is better)
* data: We load 5 urls 1 time each, and repeat for 25 cycles; collecting 25 sets of 5 data points
* summarization
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] five data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the rest; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l627 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the the subtest results.
* reporting: test time in ms (lower is better)
* contact: :jandem
* source: [https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/ARES-6 ARES-6]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: 6 cycles of the entire benchmark
** [https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#259 geometric mean] self reported from the benchmark
* contact: :davidb
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/video video]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: 12 cycles of the entire benchmark, each subtest will have 12 data points (see below)
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 24 subtest results.
* '''Lower is better'''
* unit: ms/frame
* contact: :bholley
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest perf-reftest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#Page_Load_Tests |PageLoader]]
* reporting: intervals in ms (lower is better)
* data: each test loads 25 times
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] 5 data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
* contact: :bholley
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/perf-reftest-singletons perf-reftest-singletons]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#Page_Load_Tests |PageLoader]]
* reporting: intervals in ms (lower is better)
* data: each test loads 25 times
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] 5 data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l734 source: test.py]
** suite: identical to subtest
* measuring: Time from opening a new tab (which creates a new content process) to having that new content process be ready to load URLs.
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/cpstartup cpstartup]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* bug: {{bug|1336389}}
* data: 20 cycles of the entire benchmark
* data: there are 36 reported subtests from DAMP which we load 25 times, resulting in 36 sets of 25 data points.
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l356 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 36 subtest results.
To run this locally, you'll need to pull down the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5 page set]] and run it in a local web server. See the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#tp5 |tp5 section]] or contact jmaher for more information.
{| class="mw-collapsible mw-collapsed wikitable"
* contact: :mattwoodrow
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/layout/benchmarks/displaylist_mutate.html displaylist_mutate.html]]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load the displaylist_mutate.html page five times, measuring pageload each time, generating 5 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This measures the amount of time it takes to render a page after changing its display list. The page has a large number of display list items (10,000), and mutates one every frame. The goal of the test is to make displaylist construction a bottleneck, rather than painting or other factors, and thus improvements or regressions to displaylist construction will be visible. The test runs in ASAP mode to maximize framerate, and the result is how quickly the test was able to mutate and re-paint 600 items, one during each frame.
=== Dromaeo Tests ===
Dromaeo suite of tests for JavaScript performance testing. See the [https://wiki.mozilla.org/[Dromaeo |Dromaeo wiki]] for more information.
This suite is divided into several sub-suites.
* data: Dromaeo has 6 subtests which run internal benchmarks, each benchmark reports about 180 raw data points each
summarization:
* subtest: Dromaeo is a custom benchmark which has a lot of micro tests inside each subtest, because of this we use a custom [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#dromaeo |dromaeo filter]] to summarize the subtest. Each micro test produces 5 data points and for each 5 data points we take the mean, leaving 36 data points to represent the subtest (assuming 180 points). These 36 micro test means, are then run through a geometric_mean to produce a single number for the dromaeo subtest. [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l527 source: filter.py]** suite: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 6 subtest results.
Each page in the manifest is part of the dromaeo css benchmark. Each page measures the performance of searching the DOM for nodes matching various CSS selectors, using different libraries for the selector implementation (jQuery, Dojo, Mootools, ExtJS, Prototype, and Yahoo UI).
* contact: :jgilbert
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/terrain glterrain]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load the perftest.html page (which generates 4 metrics to track) 25 times, resulting in 4 sets of 25 data points
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l381 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 4 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* contact: :jgilbert
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/webgl/benchmarks/video glvideo]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: 5 cycles of the entire benchmark, each subtest will have 5 data points (see below)
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 4; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l522 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 4 subtest results.
* '''Lower is better'''
* contact: :jandem
* source: [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/jetstream/jetstream.manifest jetstream.manifest]] and jetstream.zip from tooltool
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* measuring: JavaScript performance
* reporting: geometric mean from the benchmark
* contact: :sdetar
* source: [[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/kraken kraken.manifest]]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* measuring: JavaScript performance
* reporting: Total time for all tests, in ms (lower is better)
* data: there are 14 subtests in kraken, each subtest is an internal benchmark and generates 10 data points, or 14 sets of 10 data points.
* summarization:
** subtest: For all of the 10 data points, we take the [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#mean |mean]] to report a single number.** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 14 subtest results.
This is the [[Kraken]] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
* contact: :davidb
* source: [[https://searchfox.org/mozilla-central/source/third_party/webkit/PerformanceTests/MotionMark source]] [[https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/motionmark manifests]]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* measuring: benchmark measuring the time to animate complex scenes
* summarization:
* contact: :rhunt
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_svg.html rasterflood_svg.html]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load the rasterflood_svg.html page ten times, measuring pageload each time, generating 10 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This page animates some complex SVG patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate. The result is how quickly the browser is able to render 600 frames of the animation.
* contact: :rhunt
* source: [https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/gfx/benchmarks/rasterflood_gradient.html rasterflood_gradient.html]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load the rasterflood_gradient.html page ten times, computing a score each time, generating 10 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 9; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l986 source: test.py]
This page animates some complex gradient patterns in a requestAnimationFrame callback. However, it also churns the CPU during each callback, spinning an empty loop for 14ms each frame. The intent is that, if we consider the rasterization costs to be 0, then the animation should run close to 60fps. Otherwise it will lag. Since rasterization costs are not 0, the lower we can get them, the faster the test will run. The test runs in ASAP mode to maximize framerate.
* data: we load the session restore index page 10 times to collect 1 set of 10 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/BuildbotPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/BuildbotPerformance/Talos/Data#median |median]] of the remaining 9 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]
** suite: identical to subtest
* contact: :selena
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/speedometer/speedometer.manifest speedometer.manifest]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* measuring: responsiveness of web applications
* reporting: runs/minute score
* summarization:
** subtest: For all of the 16 subtests, we collect the sum of all their internal benchmark results.
** suite: geometric [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#mean |mean]] of the 16 sums
This is the [http://browserbench.org/Speedometer/ Speedometer] javascript benchmark taken verbatim and slightly modified to fit into our pageloader extension and talos harness.
* contact: :emilio
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/stylebench/stylebench.manifest stylebench.manifest]
* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* measuring: speed of dynamic style recalculation
* reporting: runs/minute score
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 19 data points
** suite: geometric_mean(subtests)
* data: there are 30 reported subtests from TART which we load 25 times, resulting in 30 sets of 25 data points.
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 24 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 30 subtest results.
TART is the '''Tab Animation Regression Test'''.
=== tp5 ===
* contact: :jmaher
* source: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5n.zip]]* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load each of the 51 tp5o pages 25 times, resulting in 51 sets of 25 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5n.zip]].
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l449 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 51 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
|}
Tests the time it takes Firefox to load the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5 web page test set]]. The web set was culled from the Alexa top 500 April 8th, 2011 and consists of 51 pages.
Here are the broad steps we use to create the test set:
=== tp5o_scroll ===
* contact: :kats
* source: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5n.zip]]* type: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos#Page_Load_Tests |PageLoader]]
* data: we load each of the 51 tp5o pages 12 times, resulting in 51 sets of 12 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5n.zip]].
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 11; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l470 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 51 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* contact: :rwood, :jmaher
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/tests/quantum_pageload tp6]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#Page_Load_Tests |PageLoader]]
* reporting: intervals in ms (lower is better)
* data: we load each of the tp6 pages 25 times
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first five]] data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20; [http://searchfox.org/mozilla-central/rev/8a61c71153a79cda2e1ae7d477564347c607cc5f/testing/talos/talos/test.py#246 source: test.py]
** suite: identical to subtest; each tp6_* test is reported individually (they're NOT aggregated)
The mitmproxy tool is started, the proxy setup in Firefox, and the mitmproxy https security certificate is loaded into Firefox, all automatically. The pagesets are made available for playback via the proxy, at which point each of the tp6 test pages are ready for load and measurement.
Note: For local debugging purposes, you can find the [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |page set here]]
The tp6 suite currently consists of these individual pageload tests:
* data: we load the tpaint test window 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
* data: we open a new browser window 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/BuildbotPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/BuildbotPerformance/Talos/Data#median |median]] of the remaining 15; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l190 source: test.py]
** suite: identical to subtest
* reporting:
* data: we load 50 web pages 5 times each, resulting in 50 sets of 5 data points.
* '''To run it locally''', you'd need [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#page_sets |tp5n.zip]].
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 4 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l305 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 50 subtest results.
{| class="mw-collapsible mw-collapsed wikitable"
* data: we run the tresize test page 20 times, resulting in 1 set of 20 data points.
* summarization:
** subtest: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 15 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l205 source: test.py]
** suite: same as subtest result
* summarization:
** subtest: identical to suite
** suite: [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 19 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l135 source: test.py]
Starts the browser to display tspaint_test.html with the start time in the url, waits for [[Buildbot/Talos/Tests#Paint_Tests|MozAfterPaint and onLoad]] to fire, then records the end time and calculates the time to startup.
* data: we load 6 pages 25 times each, collecting 6 sets of 25 data points
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] data point, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 24; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 6 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 2 svg opacity pages 25 times, resulting in 2 sets of 25 data points
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 2 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 5 svg pages 25 times, resulting in 5 sets of 25 data points
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 5 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* data: we load the 7 svg pages 25 times, resulting in 7 sets of 25 data points
* summarization:
** subtest: [https:/[TestEngineering/wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#ignore_first |ignore first]] '''5''' data points, then take the [https://wiki.mozilla.org[TestEngineering/Performance_sheriffingPerformance/Talos/Data#median |median]] of the remaining 20; [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/test.py#l623 source: test.py]** suite: [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#geometric_mean |geometric mean]] of the 7 subtest results.
{| cellspacing="1" cellpadding="1" border="1"
* contact: :hsivonen
* source: [https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#Microbench |Microbench]]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#median |median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
Test whose name starts with PerfIsASCII test the performance of the XPCOM string IsASCII function with ASCII inputs if different lengths.
* type: Custom GTest micro-benchmarking
* data: Time taken for a GTest function to execute
* summarization: Not a Talos test. This suite is provides a way to add low level platform performance regression tests for things that are not suited to be tested by Talos. See the [[https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Sheriffing#Microbench_Policy |Microbench Sheriffing Policy]]] for some notes on how to treat regressions.
=== PerfStrip Tests ===
* contact: :bholley, :SimonSapin
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
* type: [https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests[#Microbench |Microbench]]
* reporting: intervals in ms (lower is better)
* data: each test is run and measured 5 times
* summarization: take the [https:[TestEngineering//wiki.mozilla.org/Performance_sheriffingPerformance/Talos/Data#median |median]] of the 5 data points; [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#43-46 source: MozGTestBench.cpp]
