947
edits
Auto-tools/Projects/Signal From Noise/Execution2012: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise/Execution2012 (view source)
Revision as of 23:01, 28 December 2012
, 28 December 2012→Goals of Signal from Noise: reword
(→Goals of Signal from Noise: reword) |
|||
| Line 6: | Line 6: | ||
* Enhance fidelity of performance testing measurements: we want to ensure that (nearly) all regressions and improvements are detected. We also want to ensure that as few false positives (the noise) are reported as possible | * Enhance fidelity of performance testing measurements: we want to ensure that (nearly) all regressions and improvements are detected. We also want to ensure that as few false positives (the noise) are reported as possible | ||
* Understand our performance statistics models and assumptions end-to-end and ensure that the statistics we are using are valid | * Understand our performance statistics models and assumptions end-to-end and ensure that the statistics we are using are valid | ||
** Document and expose these | ** Document and expose these statistical methodologies in such a way that they are transparent | ||
* Make it easier/possible for developers to detect regressions in their code both locally and from [https://wiki.mozilla.org/Build:TryServer try server]. | * Make it easier/possible for developers to detect regressions in their code both locally and from [https://wiki.mozilla.org/Build:TryServer try server]. | ||
* Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | * Turn talos jobs on TBPL orange when a talos regression is detected. Developers look at https://tbpl.mozilla.org/ as the singular source of truth for what is good or bad for a given push. | ||