Confirmed users
3,339
edits
TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions
Jump to navigation
Jump to search
m
TestEngineering/Performance/Sheriffing/Workflow (view source)
Revision as of 23:55, 23 June 2023
, 23 June 2023→Identifying the culprit: Formatting and typos
m (→How to investigate an alert: typos) |
m (→Identifying the culprit: Formatting and typos) |
||
| Line 118: | Line 118: | ||
== Identifying the culprit == | == Identifying the culprit == | ||
If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here: | If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here: | ||
* | |||
* | * If the revision contains changes only from one bug, you need to open a regression bug for that. | ||
* | * If the revision contains changes from several bugs but you are familiar with the test and you know which of the bugs caused the regression, open a bug for that one. | ||
* If the revision contains changes from several bugs (usually a merge from one of the other repos), you need to do a [https://docs.google.com/document/d/1SP3rckeNOuZ6X-LzSEPjoGqQICApIzIiPlgrgJmdU38 bisection] in order to identify the causing bug. | |||
=== Harness alerts === | === Harness alerts === | ||
Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as | Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as WONTFIX and add the "harness" tag. Otherwise, open a regression bug. | ||
=== Backout/regression-fix alerts === | === Backout/regression-fix alerts === | ||
| Line 130: | Line 131: | ||
=== Infra alerts === | === Infra alerts === | ||
The | |||
The regressions caused by infra changes are probably the most difficult to identify. Excepting the case when the infra change is announced and known of, an infra regression is usually most likely to be detected by the sheriff after all the suspect commits/bugs were removed from the list. | |||
<br /> | <br /> | ||
<br /> | <br /> | ||
| Line 139: | Line 141: | ||
[[File:Grapg infra change.png|1200px|Graph infra change]]<br /> | [[File:Grapg infra change.png|1200px|Graph infra change]]<br /> | ||
<br /> | <br /> | ||
For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes | For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes related to the infrastructure that is very useful when the investigation is leading to this kind of regression: https://changelog.dev.mozaws.net/ | ||
=== Invalid alerts === | === Invalid alerts === | ||
| Line 145: | Line 147: | ||
<br /> | <br /> | ||
<br /> | <br /> | ||
In the graph below, the regression appeared around '''Dec 9''' and as you can see, there a pattern of vary predominantly between 0.7 and 1. If you click on first highlighted datapoint (around '''Dec 3''') you’ll see that its alert is marked as invalid. | In the graph below, the regression appeared around '''Dec 9''' and as you can see, there a pattern of vary predominantly between 0.7 and 1. If you click on first highlighted datapoint (around '''Dec 3'''), you’ll see that its alert is marked as invalid. | ||
<br /> | <br /> | ||
<br /> | <br /> | ||