Running Talos Tests
Prerequisite: Firefox has already been built successfully locally.
For example, to run the 'chromez' test suite:
./mach talos-test --suite chromez
To run a single test (the 'tart' test):
./mach talos-test --activeTests tart
To run two single tests, use the ':' to separate the test list:
./mach talos-test --activeTests tart:damp
Certain talos suites/tests require external pagesets to be downloaded. If pagesets are required, they will be downloaded and extracted automatically when running talos via mach. To disable this automatic pageset downloading, add the 'no-download' argument:
./mach talos-test --suite g1 --no-download
To see a list of available talos suites:
./mach talos-test --print-suites
To see a list of available talos tests:
./mach talos-test --print-tests
Further help running talos via mach:
./mach talos-test --help
Tip: Clear the operating system cache between runs, in order to significantly improve the consistency of the results. For example, on linux:
find . | xargs touch
To run talos tests on the try server, you just simply push to try as normal, specifying that you want to run Talos tests. It is highly recommended that you use trychooser to build your try server syntax (helpful hint for using trychooser: in the list of talos suites available you can hover over any one of the suites, and a list of tests included in that suite will be displayed).
To run all of the talos tests as part of your try push, just add "-t all" to the end of your try syntax. To run a single talos suite as part of your try push, just add "-t <name of suite>" to your try syntax (i.e. "-t g1").
To turn on profiling as part of your talos tests on try, add "mozharness: --spsProfile" to the end of the try syntax (i.e. "-t g1 mozharness: --spsProfile"). There is an option to do this in trychooser.
When viewing your talos try server results on Treeherder, look for the group names 'T' and 'T-e10s'. Click on the green symbol corresponding to the talos suite name, and then click on the 'Performance' tab below to see the test results.
Are my Talos Results OK?
The best way to answer this question is to push to try server and compare the performance of your push against a baseline, using Perfherder (it is recommended hitting retrigger a few times to get at least 5 data points for each test). After pushing to try with talos tests enabled, you will receive an email with a link to Perfherder where you can compare your results. Be aware of the options: e.g. pgo vs non-pgo or e10s vs non-e10s.
Hacking on Talos Itself
Local clone of mozilla-central and a successful mozilla-central Firefox build completed.
On Linux, setuptools installed:
apt-get install python-setuptools
Not required on Windows. For other operating systems see Setuptools.
On Linux, virtualenv installed:
pip install virtualenv
Not required on Windows. For other operating systems see Virtualenv.
Setup talos (linux):
cd mozilla-central/testing/talos virtualenv . . bin/activate python setup.py develop
Setup talos (Windows with mozilla-build):
cd mozilla-central/testing/talos python INSTALL.py # only required once or after updating mozilla-build source Scripts/activate # you can also use '.' (dot) instead of 'source'
Testing your Talos Patch Locally
Testing locally involves running some subset of the Talos tests on desktop and possibly mobile. Obviously not all permutations of tests and the ways of running them can be tried, so common sense should be used as to what is run. You may also want to run Talos' internal unittests: http://hg.mozilla.org/build/talos/file/tip/tests
You should tailor your choice of tests to pick those that cover what you've changed programmatically, but in general you should probably run at least one startup test and one pageloader test. A good baseline might be the 'tresize' and 'tsvgx' test suites.
Run the tests from mozilla-central/testing/talos (with the venv activated, as noted above):
talos --develop --executablePath pathtofirefox --activeTests tart
The '--develop' option indicates to run in develop mode and store the results in local.json + local.out.
The '--executablePath' option tells Talos where the firefox installation we want to run is located (i.e. '~/mozilla/objdir/dist/bin/firefox' or whatever the full path is to your firefox executable that will be running the tests).
The '--activeTests' argument provides is a list of tests we want to run (if running multiple tests, separate each test name with ':').
For a list of complete options:
Testing your Talos Patch on Try Server
Just push your talos patch to the try server. See the Try Server section above.