Buildbot/Talos: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Redirected page to Performance sheriffing/Talos)
 
(22 intermediate revisions by 10 users not shown)
Line 1: Line 1:
[[Image:Talos.jpg]]
#redirect [[Performance_sheriffing/Talos]]
 
 
== Talos ==
 
Talos is a python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.
 
So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. '''Basically, he was awesome.'''
 
Talos code lives in hg: http://hg.mozilla.org/build/talos
 
== Talos Tests ==
 
{| cellspacing="1" cellpadding="1" border="1"
|-
! [https://wiki.mozilla.org/Buildbot/Talos#NoChrome_Tests N = _nochrome]; [https://wiki.mozilla.org/Buildbot/Talos#Paint_Tests P = _paint]
!colspan=5|TBPL Abbreviation (TestName)
|-
! TestName
! Trunk
! Aurora
! Beta
! Release 
! ESR
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tp5 tp5]]
| tp (tp5n)
| tp (tp5n)
| tp (tp5n)
| tp (tp5row)
| tp (tp5)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tsvg tsvg]]
| s (tsvgr)
| s (tsvgr)
| s (tsvgr) 
| s (tsvg) 
| s (tsvg)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tsvg-opacity tsvg_opacity]]
| s (tsvgr_opacity)
| s (tsvgr_opacity)
| s (tsvgr_opacity)
| s (tsvg_opacity)
| s (tsvg_opacity)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tdhtml tdhtml]]
| deactivated
| deactivated
| deactivated
| deactivated
| deactivated
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tdhtml tdhtml_nochrome]]
| deactivated
| deactivated
| deactivated
| deactivated
| deactivated
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#a11y a11y]]
| o (a11yr P)
| o (allyr P)
| o (allyr P)
| c (ally.2 P) 
| c (ally P)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#ts_paint ts_paint]]
| o (ts_paint)
| o (ts_paint)
| o (ts_paint)
| c (ts_paint)
| c (ts_paint)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tpaint tpaint]] (aka twinopen/txul)
| o (tpaint)
| o (tpaint)
| o (tpaint)
| c (tpaint)
| c (tpaint)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#Dromaeo_CSS dromaeo_css]]
| d
| d
| d
| dr
| dr
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#Dromaeo_DOM dromaeo_dom]]
| d
| d
| d
| dr
| dr
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#sunspider tsspider]]
| deactivated
| deactivated
| deactivated
| c (tsspider.2 P) 
| c (tsspider P)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tsspider tsspider_nochrome]]
| deactivated
| deactivated
| deactivated
| n (tsspider.2 NP)
| n (tsspider NP)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#xperf xperf]]
| x (windows only)
|
|
| 
|
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#ts_places_generated_med ts_places_generated_med]]
| p (P)
| p (P)
| p (P)
| di
| di
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#ts_places_generated_max ts_places_generated_max]]
| p (P)
| p (P)
| p (P)
| di
| di
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tscroll tscroll]]
| o (tscrollr)
| o (tscrollr)
| o (tscrollr)
| c (tscroll.2)
| c (tscroll)
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#tresize tresize]]
| c (tresize)
| c (tresize)
| c (tresize)
|
|
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#sunspider sunspider 0.9.1]]
| d (sunspider)
| d (sunspider)
| d (sunspider)
|
|
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#kraken kraken]]
| d (kraken)
| d (kraken)
| d (kraken)
|
|
|-
| [[https://wiki.mozilla.org/Buildbot/Talos/Tests#V8 v8 (version 7)]]
| d (v8_7)
| d (v8_7)
| d (v8_7)
|
|
|}
 
More information is available in [[Buildbot/Talos/Tests]] or by using the command line switch: ''talos --print-tests''
 
== Lifecycle of a Performance Test, from push to graph server ==
 
* Buildbot schedules Talos jobs
* Buildbot runs a talos job on a slave when one is available
** Buildbot downloads the build and talos.zip (found in [http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json talos.json])
** Buildbot [https://wiki.mozilla.org/Buildbot/Talos/Running#How_Talos_is_Run_in_Production configures the test and runs it]
** After the test is completed the data is uploaded to the [http://graphs.mozilla.org/ Graph Server] (and to [https://datazilla.mozilla.org/talos/ Datazilla])
** a link to the Graph Server data (ex. [http://graphs.mozilla.org/graph.html#tests=[[206,63,22]] tp5]) is posted in the log file
* TBPL displays a green (all OK) status and has a link to the graph server
* 5 pushes later, [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] is ran which compares your push to the previous 30 pushes and next 5 pushes to look for a [https://wiki.mozilla.org/Buildbot/Talos/Data#Regressions regression]
** if a regression is found, it will be posted on [http://groups.google.com/group/mozilla.dev.tree-management/topics?lnk=srg&pli=1 dev.tree-management]
 
== Links ==
 
=== Signal From Noise ===
 
The [https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise Signal from Noise] project is an effort to increase the fidelity of regression detection for performance tests.
 
=== Adding a new test ===
If you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the [https://wiki.mozilla.org/Buildbot/Talos/Misc#Adding_a_new_test steps required].
 
=== Bugs ===
* [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Talos&product=Testing Talos bugs] are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos&op_sys=All&rep_platorm=All Testing/Talos], such as requests for new tests or repairs to the talos code itself.
* Graph server bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=webtools Webtools/Graph server].
* Datazilla bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Datazilla Datazilla].
* Talos machine maintenance bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org mozilla.org/Release Engineering], such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.
 
=== Subpages ===
* Definition of [https://wiki.mozilla.org/Buildbot/Talos/Tests tests]
* How to [https://wiki.mozilla.org/Buildbot/Talos/Running Run Talos], and how it is run in production.
** How to run Talos on [[Mobile/Fennec/Android#talos mobile]]
* [https://wiki.mozilla.org/Buildbot/Talos/Internals Internals] of Talos
** [Buildbot/Talos/Counters] : Talos counters and auxiliary data '''(STUB)'''
* Where the [https://wiki.mozilla.org/Buildbot/Talos/Data data] is stored and how it is calculated
** Specifics of the [https://wiki.mozilla.org/Buildbot/Talos/DataFormat Data Format]
* History and other uses of [https://wiki.mozilla.org/Buildbot/Talos/Misc Talos]
 
=== Blog posts about Talos ===
* https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
* https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/
* http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html

Latest revision as of 09:12, 7 August 2018

Redirect to:

Retrieved from "https://wiki.mozilla.org/index.php?title=Buildbot/Talos&oldid=1198534"