Buildbot/Talos

Talos

Talos is a python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.

So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. Basically, he was awesome.

Talos Tests

N = _nochrome; P = _paint	TBPL Abbreviation (TestName)
TestName	Trunk	Aurora	Beta	Release	ESR
[tp5]	tp (tp5n)	tp (tp5n)	tp (tp5n)	tp (tp5row)	tp (tp5)
[tsvg]	s (tsvgr)	s (tsvgr)	s (tsvgr)	s (tsvg)	s (tsvg)
[tsvg_opacity]	s (tsvgr_opacity)	s (tsvgr_opacity)	s (tsvgr_opacity)	s (tsvg_opacity)	s (tsvg_opacity)
[tdhtml]	deactivated	deactivated	deactivated	deactivated	deactivated
[tdhtml_nochrome]	deactivated	deactivated	deactivated	deactivated	deactivated
[a11y]	o (a11yr P)	o (allyr P)	o (allyr P)	c (ally.2 P)	c (ally P)
[ts_paint]	o (ts_paint)	o (ts_paint)	o (ts_paint)	c (ts_paint)	c (ts_paint)
[tpaint] (aka twinopen/txul)	o (tpaint)	o (tpaint)	o (tpaint)	c (tpaint)	c (tpaint)
[dromaeo_css]	d	d	d	dr	dr
[dromaeo_dom]	d	d	d	dr	dr
[tsspider]	deactivated	deactivated	deactivated	c (tsspider.2 P)	c (tsspider P)
[tsspider_nochrome]	deactivated	deactivated	deactivated	n (tsspider.2 NP)	n (tsspider NP)
[xperf]	x (windows only)
[ts_places_generated_med]	p (P)	p (P)	p (P)	di	di
[ts_places_generated_max]	p (P)	p (P)	p (P)	di	di
[tscroll]	o (tscrollr)	o (tscrollr)	o (tscrollr)	c (tscroll.2)	c (tscroll)
[tresize]	c (tresize)	c (tresize)	c (tresize)
[sunspider 0.9.1]	d (sunspider)	d (sunspider)	d (sunspider)
[kraken]	d (kraken)	d (kraken)	d (kraken)
[v8 (version 7)]	d (v8_7)	d (v8_7)	d (v8_7)

Lifecycle of a Performance Test, from push to graph server

• Buildbot schedules Talos jobs
• Buildbot runs a talos job on a slave when one is available
  • Buildbot downloads the build and talos.zip (found in talos.json)
  • Buildbot configures the test and the runs it
  • After the test is completed the data is uploaded to the Graph Server (and to Datazilla)
  • a link to the Graph Server data (ex. tp5) is posted in the log file
• TBPL displays a green (all OK) status and has a link to the graph server
• 5 pushes later, analyze_talos.py is ran which compares your push to the previous 30 pushes and next 5 pushes to look for a regression
  • if a regression is found, it will be posted on dev.tree-management

Useful Links

• Talos code lives in hg: http://hg.mozilla.org/build/talos
• Running Talos on Mobile/Fennec/Android#talos mobile

Adding a new test

If you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the steps required.

Bugs

• Talos bugs are filed under Testing/Talos, such as requests for new tests or repairs to the talos code itself.
• Graph server bugs are filed under Webtools/Graph server.
• Datazilla bugs are filed under Datazilla.
• Talos machine maintenance bugs are filed under mozilla.org/Release Engineering, such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.

Subpage

• Definition of tests
• How to Run Talos, and how it is run in production.
• Internals of Talos
• Where the data is stored and how it is calculated
  • Specifics of the [Data Format]
• History and other uses of Talos

Blog posts about Talos

• https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
• https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/