Buildbot/Talos
Talos
Talos is a python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.
So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. Basically, he was awesome.
Talos Tests
| N = _nochrome; P = _paint | TBPL Abbreviation (TestName) | ||||
|---|---|---|---|---|---|
| TestName | Trunk | Aurora | Beta | Release | ESR |
| [tp5] | tp (tp5n) | tp (tp5n) | tp (tp5n) | tp (tp5row) | tp (tp5) |
| [tsvg] | s (tsvgr) | s (tsvgr) | s (tsvgr) | s (tsvg) | s (tsvg) |
| [tsvg_opacity] | s (tsvgr_opacity) | s (tsvgr_opacity) | s (tsvgr_opacity) | s (tsvg_opacity) | s (tsvg_opacity) |
| [tdhtml] | deactivated | deactivated | deactivated | deactivated | deactivated |
| [tdhtml_nochrome] | deactivated | deactivated | deactivated | deactivated | deactivated |
| [a11y] | o (a11yr P) | o (allyr P) | o (allyr P) | c (ally.2 P) | c (ally P) |
| [ts_paint] | o (ts_paint) | o (ts_paint) | o (ts_paint) | c (ts_paint) | c (ts_paint) |
| [tpaint] (aka twinopen/txul) | o (tpaint) | o (tpaint) | o (tpaint) | c (tpaint) | c (tpaint) |
| [dromaeo_css] | d | d | d | dr | dr |
| [dromaeo_dom] | d | d | d | dr | dr |
| [tsspider] | deactivated | deactivated | deactivated | c (tsspider.2 P) | c (tsspider P) |
| [tsspider_nochrome] | deactivated | deactivated | deactivated | n (tsspider.2 NP) | n (tsspider NP) |
| [xperf] | x (windows only) | ||||
| [ts_places_generated_med] | p (P) | p (P) | p (P) | di | di |
| [ts_places_generated_max] | p (P) | p (P) | p (P) | di | di |
| [tscroll] | o (tscrollr) | o (tscrollr) | o (tscrollr) | c (tscroll.2) | c (tscroll) |
| [tresize] | c (tresize) | c (tresize) | c (tresize) | ||
| [sunspider 0.9.1] | d (sunspider) | d (sunspider) | d (sunspider) | ||
| [kraken] | d (kraken) | d (kraken) | d (kraken) | ||
| [v8 (version 7)] | d (v8_7) | d (v8_7) | d (v8_7) | ||
Lifecycle of a Performance Test, from push to graph server
- TODO an abbreviated version of this.
Adding a new test
Everybody wants moar tests, there is a lot that goes into adding a talos test:
- file a bug to add appropriate rows to graphserver.
- file an additional IT bug to deploy the sql changes on the production and staging graph servers
- if this is adding new pages, ensure these sql changes include page definitions. NOTE: this one detail is usually forgotten since it is so rare
- file a bug to add tests to talos.
- create a talos.zip file and file a releng bug to upload it to the build network
- create a patch for buildbot to add a definition of this new test and turn it on for the current branch
- create a m-c (not inbound) patch to modify testing/talos/talos.json, get it reviewed and landed
- create bugs for each time we uplift from mozilla-central->aurora->beta->release->esr to turn on your test
- if this is an update to an existing test that could change the numbers, this needs to be treated as a new test and run side by side for a week to get a new baseline for the numbers.
- file a bug to get tbpl updated with a new letter to track this test
While that is a laundry list of items to do, if you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the steps above. When adding a new test, we really need to understand what we are doing. Here are some questions that you should know the answer to before adding a new test:
- What does this test measure?
- Does this test overlap with any existing test?
- What is the unit of measurement that we are recording?
- What would constitute a regression?
- What is the expected range in the results over time?
- Are there variables or conditions which would affect this test?
- browser configuration (prefs, environment variables)?
- OS, resources, time of day, etc... ?
- Indepenedent of Observation? Will this test produce the same number regardless of what was run before it?
- What considerations are there for how this test should be run and what tools are required?
Bugs
- Talos bugs are filed under Testing/Talos, such as requests for new tests or repairs to the talos code itself.
- Graph server bugs are filed under Webtools/Graph server.
- Datazilla bugs are filed under Datazilla.
- Talos machine maintenance bugs are filed under mozilla.org/Release Engineering, such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.
Useful Links
- Talos code lives in hg: http://hg.mozilla.org/build/talos
- Running Talos on Mobile/Fennec/Android#talos mobile
- Subpages: