Confirmed users
3,376
edits
Auto-tools/Projects/Signal From Noise: Difference between revisions
Jump to navigation
Jump to search
Auto-tools/Projects/Signal From Noise (view source)
Revision as of 17:26, 6 January 2012
, 6 January 2012→Use Cases
No edit summary |
|||
| Line 122: | Line 122: | ||
=== Use Cases === | === Use Cases === | ||
* developer: push patch to mozilla-central, | Currently: | ||
* Firefox developer: | |||
** push patch to mozilla-central, expect green talos results | |||
*** all results are green unless test fails to complete, then it is red | |||
** notification to dev.tree-management indicates a regression | |||
*** developer goes to graphs-new and looks at the (test, platform, branch) graph | |||
*** maybe compares to other platforms or branches | |||
* Talos developer | |||
** adds new feature to talos with expected change in numbers | |||
** run change side by side as a new test name for 1 week | |||
** browse to graphs-new to view new_test vs old_test to look at raw data points over a few days on each platform | |||
Proposed 1 (assuming 1% deviation): | |||
* Firefox developer | |||
** push patch to mozilla-central, expect green talos results | |||
*** number outside of 2% from gold standard, run turns orange | |||
*** orange run has link on tbpl to graph server | |||
*** graph server has a quick line of historical data and other platforms | |||
*** then a focused section of what the gold standard is and what that run produced | |||
*** it would be nice to see what the previous 5 runs had in terms of numbers, as well as all other platforms | |||
** no need for notification mails to dev.tree-management since this is managed in tbpl | |||
** FLAW: if firefox adjusts the standard number (up or down), then how do we call it the new standard? | |||
*** maybe the web interface can have a way to change the number on the fly and put a bug/comment for the adjustment | |||
* Talos developer | |||
** adds new feature to talos with expected change in numbers | |||
** while pushing add an entry to the graph server of the new expected number | |||
** no need for side by side since we are just comparing to a known standard number | |||