Locally

Warning: Firefox must have already been built successfully locally. See Building Firefox for details.

mach is used to run the Talos tests locally. You can run suites of tests or individual tests on their own.

For example, to run the 'chromez' test suite:

 ./mach talos-test --suite chromez

To run a single test (the 'tart' test):

 ./mach talos-test --activeTests tart

Known Issues

In some cases, the Talos setup may not finish properly, throwing Import Error: No module named for packages such as wptserve and simplejson. This is known to affect MacOS users: Bug 1667189 - Talos test refuses to run because of missing python packages

If you encounter this error, running the following command should resolve it:

 brew link python3

Otherwise, you can manually activate Talos's virtual environment (found in your obj directory, which differs based on your machine architecture and OS, i.e. obj-x86_64-apple-darwin19.6.0) and install the missing packages:

 source mozilla-central/obj-[your directory suffix]/testing/talos-venv/bin/activate
 pip install wptserve simplejson

Certain Talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running Talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument:

 ./mach talos-test --suite g1 --no-download

To see a list of available Talos suites:

 ./mach talos-test --print-suites

To see a list of available Talos tests:

 ./mach talos-test --print-tests

Further help running talos via mach:

 ./mach talos-test --help

Tip: Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux:

 find . | xargs touch

Try server

Note: It's recommended to run ~5 times using the --rebuild option due to the noise inherent in performance test results

Talos tests can be triggered using the try server. The following command will run all Talos tests against all available platforms five times using the fuzzy selector:

 ./mach try fuzzy --query "^test- 'talos" --rebuild 5

To turn on profiling as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.

When viewing your Talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the Talos suite name, and then click on the 'Performance' tab below to see the test results.

If you're curious about the hardware we use, here is a description of the specs. You can also refer to the test reference platforms documentation.

Are my Talos results OK?

The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with Talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Make sure you look at the different platforms.