Auto-tools/Projects/Signal From Noise/StatusNovember2011: Difference between revisions

Jump to navigation Jump to search

Auto-tools/Projects/Signal From Noise/StatusNovember2011 (view source)

Revision as of 22:20, 28 December 2012

2 bytes removed ,  28 December 2012
m
→‎Review of Current Performance Testing Workflow: tense
(→‎Review of Current Performance Testing Workflow: link to extant regression methodology)
Line 11: Line 11:
* a script versioned with graphserver (but unrelated to and unused by graphs.mozilla.org) mails dev-tree-management when a regression or improvement is detected: http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py . The methodology used is available from inspection of the script or https://wiki.mozilla.org/images/c/c0/Larres-thesis.pdf#page=73
* a script versioned with graphserver (but unrelated to and unused by graphs.mozilla.org) mails dev-tree-management when a regression or improvement is detected: http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py . The methodology used is available from inspection of the script or https://wiki.mozilla.org/images/c/c0/Larres-thesis.pdf#page=73


It is a common misconception that [https://tbpl.mozilla.org/ TBPL] talos tests turned orange for test failure.  In fact, as long as no infrastructure problems are encountered, Talos runs always report green regardless of the performance of the system under test.  Graphserver and the talos regression emails are the only source of information as to whether a regression or improvement has resulted.  However, the existing methods of calculating statistics (professing to detect regressions as low as 1%)  lead to a high degree of noise.  The consequence is confusion amongst developers with regard to whether a reported regression is legitimate or not; the consequence of this is that the regression emails are largely ignored since there are so many of them and most of them are merely noise.
It is a common misconception that [https://tbpl.mozilla.org/ TBPL] talos tests turn orange for test failure.  In fact, as long as no infrastructure problems are encountered, Talos runs always report green regardless of the performance of the system under test.  Graphserver and the talos regression emails are the only source of information as to whether a regression or improvement has resulted.  However, the existing methods of calculating statistics (professing to detect regressions as low as 1%)  lead to a high degree of noise.  The consequence is confusion amongst developers with regard to whether a reported regression is legitimate or not; the consequence of this is that the regression emails are largely ignored since there are so many of them and most of them are merely noise.


It is another common misconception that all of the "Talos regression/improvement" emails actually come from the Talos tests.  Some (e.g. numbers of constructors) do not.
It is another common misconception that all of the "Talos regression/improvement" emails actually come from the Talos tests.  Some (e.g. numbers of constructors) do not.
Jhammel
947

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/495878"

Navigation menu