Confirmed users
2,197
edits
TestEngineering/Performance/Sheriffing: Difference between revisions
TestEngineering/Performance/Sheriffing (view source)
Revision as of 11:44, 16 March 2021
, 16 March 2021no edit summary
Alexandrui (talk | contribs) |
No edit summary |
||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 3: | Line 3: | ||
A Performance Sheriff's job is similar: to make sure that performance regressions in Firefox are detected and dealt with. They look at the data produced by performance test jobs (mainly Talos), find regressions, determine their root cause(s) and gets bugs on file to track all issues and make interested parties aware of what is going on. | A Performance Sheriff's job is similar: to make sure that performance regressions in Firefox are detected and dealt with. They look at the data produced by performance test jobs (mainly Talos), find regressions, determine their root cause(s) and gets bugs on file to track all issues and make interested parties aware of what is going on. | ||
A short demo describing the performance sheriffing workflow: | |||
https://youtu.be/VdTC7Maa6hE | |||
= What is an alert = | = What is an alert = | ||
| Line 11: | Line 14: | ||
** [[/Tree_FAQ#Branch_names_and_confusion|branch]] | ** [[/Tree_FAQ#Branch_names_and_confusion|branch]] | ||
** % regressed, this is a range of the regressions (not improvements) | ** % regressed, this is a range of the regressions (not improvements) | ||
** the [ | ** the [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions tests] which have regressed | ||
** the platforms we see this regression on | ** the platforms we see this regression on | ||
* date of the suspect revision push | * date of the suspect revision push | ||
| Line 19: | Line 22: | ||
Below the summary will be a list of alerts, each alert will reference: | Below the summary will be a list of alerts, each alert will reference: | ||
* [ | * [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Test name] | ||
* platform (including build type, such as opt, pgo) | * platform (including build type, such as opt, pgo) | ||
* old score (median score of the previous 12 commits) | * old score (median score of the previous 12 commits) | ||
| Line 98: | Line 101: | ||
* Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos] | * Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos] | ||
* Dependent/Block bugs - For a new bug, add the [[#Tracking_bugs|tracking bug]] (for the current version) and root cause bug(s) as blocking this bug | * Dependent/Block bugs - For a new bug, add the [[#Tracking_bugs|tracking bug]] (for the current version) and root cause bug(s) as blocking this bug | ||
* CC list - cc patch author(s), reviewer(s) and owner of the tests as documented | * CC list - cc patch author(s), reviewer(s) and owner of the tests as documented in the [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Talos test definition]; if we have >1 bug, we should cc everyone who worked on those bugs so we call pitch in an answer questions faster | ||
* Summary of bug should have a check to make sure the revision is accurate | * Summary of bug should have a check to make sure the revision is accurate | ||
* The description is auto suggested as well, please verify the revision here | * The description is auto suggested as well, please verify the revision here | ||
| Line 125: | Line 128: | ||
= Additional Resources = | = Additional Resources = | ||
* [[TestEngineering/Performance/ | * [[TestEngineering/Performance/FAQ]] | ||