Confirm
2,177
edits
Changes
Added notifications of content migration to Firefox Source Docs
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html|date=2021-03-16}}
= At a glance =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#at-a-glance|date=2021-03-16|section=yes}}
* Tests are defined in [https://searchfox.org/mozilla-central/source/testing/talos/talos/test.py testing/talos/talos/test.py]
* Treeherder abbreviations are defined in [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml taskcluster/ci/test/talos.yml]
= Test lifecycle =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-lifecycle|date=2021-03-16|section=yes}}
* Taskcluster schedules [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml talos jobs]
* Taskcluster runs a Talos job on a hardware machine when one is available - this is bootstrapped by [https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py mozharness]
= Test types =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-types|date=2021-03-16|section=yes}}
There are two different species of Talos tests:
== Startup ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#startup|date=2021-03-16|section=yes}}
[https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test Startup tests] launch Firefox and measure the time to the onload or paint events. We run this in a series of cycles (default to 20) to generate a full set of data. Tests that currently are startup tests are:
* [[#ts_paint]]
== Page load ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#page-load|date=2021-03-16|section=yes}}
Many of the talos tests use the page loader to load a manifest of pages.
These are tests that load a specific page and measure the time it takes to load the page, scroll the page, draw the page etc. In order to run a page load test, you need a manifest of pages to run. The manifest is simply a list of URLs of pages to load, separated by carriage returns, e.g.:
== Heavy ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#heavy|date=2021-03-16|section=yes}}
All our testing is done with empty blank profiles, this is not ideal for finding issues for end users. We recently undertook a task to create a daily update to a profile so it is modern and relevant. It browses a variety of web pages, and have history and cache to give us a more realistic scenario.
== Web extension ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#web-extension|date=2021-03-16|section=yes}}
Web Extensions are what Firefox has switched to and there are different code paths and APIs used vs addons. Historically we don't test with addons (other than our test addons) and are missing out on common slowdowns. In 2017 we started running some startup and basic pageload tests with a web extension in the profile ({{bug|1398974}}). We have updated the Extension to be more real world and will continue to do that.
== Real-world WebExtensions ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#real-world-webextensions|date=2021-03-16|section=yes}}
We've added a variation on our test suite that automatically downloads, installs and enables 5 popular WebExtensions. This is used to measure things like the impact of real-world WebExtensions on start-up time.
== Paint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#paint|date=2021-03-16|section=yes}}
Paint tests are measuring the time to receive both the [https://developer.mozilla.org/en-US/docs/Web/Events/MozAfterPaint MozAfterPaint] and OnLoad event instead of just the OnLoad event. Most tests now look for this unless they are an ASAP test, or an internal benchmark
== ASAP ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#asap|date=2021-03-16|section=yes}}
We have a variety of tests which we now run in ASAP mode where we render as fast as possible (disabling vsync and letting the rendering iterate as fast as it can using `requestAnimationFrame`). In fact we have replaced some original tests with the 'x' versions to make them measure. We do this with RequestAnimationFrame().
== Benchmarks ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#benchmarks|date=2021-03-16|section=yes}}
Many tests have internal benchmarks which we report as accurately as possible. These are the exceptions to the general rule of calculating the suite score as a geometric mean of the subtest values (which are median values of the raw data from the subtests).
== Row major vs. column major ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#row-major-vs-column-major|date=2021-03-16|section=yes}}
To get more stable numbers, tests are run multiple times. There are two ways that we do this: row major and column major. Row major means each test is run multiple times and then we move to the next test (and run it multiple times). Column major means that each test is run once one after the other and then the whole sequence of tests is run again.
= Page sets =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#page-sets|date=2021-03-16|section=yes}}
We run our tests 100% offline, but serve pages via a webserver. Knowing this we need to store and make available the offline pages we use for testing.
== tp5pages ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tp5pages|date=2021-03-16|section=yes}}
Some tests make use of a set of 50 "real world" pages, known as the tp5n set. These pages are not part of the talos repository, but without them the tests which use them won't run.
* To add these pages to your local setup, download the latest tp5n zip from [https://mozilla-releng.net/tooltool/ tooltool], and extract it such that `'''tp5n'''` ends up as `testing/talos/talos/tests/'''tp5n'''`. You can also obtain it by running a talos test locally to get the zip into `testing/talos/talos/tests/`, i.e <tt>./mach talos-test --suite damp</tt>
= Test definitions =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions|date=2021-03-16|section=yes}}
'''Please keep these in alphabetical order'''
== a11y ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#a11yr|date=2021-03-16|section=yes}}
'''Test Name'''
== about-newtab ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#about-newtab-with-snipppets|date=2021-03-16|section=yes}}
'''Test Name'''
== about-preferences ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#about-preferences-basic|date=2021-03-16|section=yes}}
'''Test Name'''
== ARES-6 ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#ares6|date=2021-03-16|section=yes}}
'''Test Name'''
== Basic compositor video ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#basic-compositor-video|date=2021-03-16|section=yes}}
'''Test Name'''
== cpstartup ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#cpstartup|date=2021-03-16|section=yes}}
'''Test Name'''
== DAMP ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#damp|date=2021-03-16|section=yes}}
'''Test Name'''
== displaylist_mutate ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#displaylist-mutate|date=2021-03-16|section=yes}}
'''Test Name'''
== Dromaeo ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#dromaeo|date=2021-03-16|section=yes}}
Dromaeo suite of tests for JavaScript performance testing. See the [[Dromaeo|Dromaeo wiki]] for more information.
=== Dromaeo CSS ===
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#dromaeo-css|date=2021-03-16|section=yes}}
'''Test Name'''
=== Dromaeo DOM (Linux64 only) ===
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#dromaeo-dom|date=2021-03-16|section=yes}}
'''Test Name'''
== glterrain ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#glterrain|date=2021-03-16|section=yes}}
'''Test Name'''
== glvideo ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#glvideo|date=2021-03-16|section=yes}}
'''Test Name'''
== jetstream ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#jetstream|date=2021-03-16|section=yes}}
'''Test Name'''
== kraken ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#kraken|date=2021-03-16|section=yes}}
'''Test Name'''
== motionmark_animometer ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#motionmark-animometer|date=2021-03-16|section=yes}}
'''Test Name'''
== motionmark_html ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#motionmark-html-suite|date=2021-03-16|section=yes}}
'''Test Name'''
== motionmark_webgl ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#motionmark-webgl|date=2021-03-16|section=yes}}
'''Test Name'''
== pdfpaint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#pdfpaint|date=2021-03-16|section=yes}}
'''Test Name'''
== perf-reftest ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#perf-reftest|date=2021-03-16|section=yes}}
'''Test Name'''
== perf-reftest-singletons ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#perf-reftest-singletons|date=2021-03-16|section=yes}}
'''Test Name'''
== rasterflood_gradient ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#rasterflood-gradient|date=2021-03-16|section=yes}}
'''Test Name'''
== rasterflood_svg ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#rasterflood-svg|date=2021-03-16|section=yes}}
'''Test Name'''
== sessionrestore ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#sessionrestore|date=2021-03-16|section=yes}}
'''Test Name'''
== sessionrestore_many_windows ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#sessionrestore-many-windows|date=2021-03-16|section=yes}}
'''Test Name'''
== sessionrestore_no_auto_restore ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#a11yr|date=2021-03-16|section=yes}}
'''Test Name'''
== startup_about_home_paint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#startup-about-home-paint|date=2021-03-16|section=yes}}
'''Test Name'''
</pre>
|}
== startup_about_home_paint_cached ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#startup-about-home-paint-cached|date=2021-03-16|section=yes}}
'''Test Name'''
== startup_about_home_paint_realworld_webextensions ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#startup-about-home-paint-realworld-webextensions|date=2021-03-16|section=yes}}
'''Test Name'''
== stylebench ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#stylebench|date=2021-03-16|section=yes}}
'''Test Name'''
== tabpaint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tabpaint|date=2021-03-16|section=yes}}
'''Test Name'''
== tabswitch ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tabswitch|date=2021-03-16|section=yes}}
'''Test Name'''
== tart ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tart|date=2021-03-16|section=yes}}
'''Test Name'''
== tart_flex ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tart-flex|date=2021-03-16|section=yes}}
'''Test Name'''
== tp5 ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tp5|date=2021-03-16|section=yes}}
Note that the tp5 test no longer exists (only talos-tp5o) though many tests still make use of this pageset. Here, we provide an overview of the tp5 pageset and some information about how data using the tp5 pageset might be used in various suites.
== tp5o_scroll ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tp5o-scroll|date=2021-03-16|section=yes}}
'''Test Name'''
== tp5o_webext ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tp5o-webext|date=2021-03-16|section=yes}}
'''Test Name'''
== tpaint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tpaint|date=2021-03-16|section=yes}}
{{warning|This test no longer exists}}
== tresize ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tresize|date=2021-03-16|section=yes}}
* contact: :jimm
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/startup_test/tresize/addon/content/tresize-test.html tresize-test.html]
== ts_paint ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#ts-paint|date=2021-03-16|section=yes}}
'''Test Name'''
== ts_paint_flex ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#ts-paint-flex|date=2021-03-16|section=yes}}
'''Test Name'''
== ts_paint_heavy ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#ts-paint-heavy|date=2021-03-16|section=yes}}
'''Test Name'''
== ts_paint_webext ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#ts-paint-webext|date=2021-03-16|section=yes}}
'''Test Name'''
== tscrollx ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tscrollx|date=2021-03-16|section=yes}}
'''Test Name'''
== tsvg_static ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tsvg-static|date=2021-03-16|section=yes}}
'''Test Name'''
== tsvgm ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tsvgm|date=2021-03-16|section=yes}}
'''Test Name'''
== tsvgr_opacity ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tsvgr-opacity|date=2021-03-16|section=yes}}
'''Test Name'''
== tsvgx ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#tsvgx|date=2021-03-16|section=yes}}
'''Test Name'''
== twinopen ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#twinopen|date=2021-03-16|section=yes}}
'''Test Name'''
== xperf (tp5n) ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#xperf|date=2021-03-16|section=yes}}
* contact: [mailto:fx-perf@mozilla.com fx-perf@mozilla.com]
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/talos/talos/xtalos xperf instrumentation]
= Build metrics =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#build-metrics|date=2021-03-16|section=yes}}
These are not part of the Talos code, but like Talos they are benchmarks that record data using the graphserver and are analyzed by the same scripts for regressions.
= Platform microbenchmark =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#platform-microbenchmark|date=2021-03-16|section=yes}}
== IsASCII and IsUTF8 gtest microbenchmarks ==
== Microbench ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#microbench|date=2021-03-16|section=yes}}
* contact: :bholley
* source: [https://dxr.mozilla.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp MozGTestBench.cpp]
== PerfStrip Tests ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#perfstrip-tests|date=2021-03-16|section=yes}}
* contact: :davidb
* source: https://dxr.mozilla.org/mozilla-central/source/xpcom/tests/gtest/TestStrings.cpp
== Stylo gtest microbenchmarks ==
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#stylo-gtest-microbenchmarks|date=2021-03-16|section=yes}}
* contact: :bholley, :SimonSapin
* source: [https://dxr.mozilla.org/mozilla-central/source/layout/style/test/gtest]
= History of tp tests =
{{delete|Content has been migrated to https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#history-of-tp-tests|date=2021-03-16|section=yes}}
== tp ==
