342
edits
TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions
TestEngineering/Performance/Sheriffing/Workflow (view source)
Revision as of 16:40, 9 March 2022
, 9 March 2022→Handling alerts
Alexandrui (talk | contribs) |
Alexandrui (talk | contribs) |
||
| Line 105: | Line 105: | ||
* talos, raptor, Build metrics >= 2% | * talos, raptor, Build metrics >= 2% | ||
<br/> | <br/> | ||
== Regressions vs improvements == | |||
Whilst the difference between regressions and improvements is self-explanatory, after acknowledging the alert: | |||
* the regressions go through multiple statuses until the final resolution | |||
* the improvements have the only status of "improvement" | |||
== Identifying the culprit == | == Identifying the culprit == | ||
If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here: | If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here: | ||
| Line 111: | Line 115: | ||
* The revision contains changes from several bugs (usually a merge from one of the other repos) when you need to do a [https://docs.google.com/document/d/1SP3rckeNOuZ6X-LzSEPjoGqQICApIzIiPlgrgJmdU38 bisection] in order to identify the causing bug. | * The revision contains changes from several bugs (usually a merge from one of the other repos) when you need to do a [https://docs.google.com/document/d/1SP3rckeNOuZ6X-LzSEPjoGqQICApIzIiPlgrgJmdU38 bisection] in order to identify the causing bug. | ||
== Harness alerts == | === Harness alerts === | ||
TODO: caused by recordings changing the baseline, link to culprit and close as WONTFIX | TODO: caused by recordings changing the baseline, link to culprit and close as WONTFIX | ||
== Backout/regression-fix alerts == | === Backout/regression-fix alerts === | ||
TODO: caused by recordings changing the baseline, link to culprit and close as WONTFIX | TODO: caused by recordings changing the baseline, link to culprit and close as WONTFIX | ||
== Infra alerts == | === Infra alerts === | ||
The infra regressions are caused by infra changes are probably the most difficult to identify. Excepting the case when the infra change is announced and known of, usually an infra regression is most likely to be detected by the sheriff after all the suspect commits/bugs were removed from the list. | The infra regressions are caused by infra changes are probably the most difficult to identify. Excepting the case when the infra change is announced and known of, usually an infra regression is most likely to be detected by the sheriff after all the suspect commits/bugs were removed from the list. | ||
<br /> | <br /> | ||
| Line 129: | Line 133: | ||
For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes realted to the infrastructure that is very useful when the investigation is leading to this kind of regression: https://changelog.dev.mozaws.net/ | For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes realted to the infrastructure that is very useful when the investigation is leading to this kind of regression: https://changelog.dev.mozaws.net/ | ||
== Invalid alerts == | === Invalid alerts === | ||
Invalid regressions usually (but not only) happen when the test results are very unstable. A useful tip of finding invalid regressions is looking at graph’s history for a pattern in the evolution of the datapoints. | Invalid regressions usually (but not only) happen when the test results are very unstable. A useful tip of finding invalid regressions is looking at graph’s history for a pattern in the evolution of the datapoints. | ||
<br /> | <br /> | ||