342
edits
TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions
TestEngineering/Performance/Sheriffing/Workflow (view source)
Revision as of 08:10, 27 September 2023
, 27 September 2023→Finding the culprit
Alexandrui (talk | contribs) |
Alexandrui (talk | contribs) |
||
| Line 81: | Line 81: | ||
After the jobs are finished running, you have jobs triggered on successive revisions and you can continue looking for the culprit.<br /> | After the jobs are finished running, you have jobs triggered on successive revisions and you can continue looking for the culprit.<br /> | ||
== Finding the culprit == | == Finding the culprit == | ||
A clear improvement/regression appears usually when there is easily noticeable difference between two adjacent data points: | * A clear improvement/regression appears usually when there is easily noticeable difference between two adjacent data points: | ||
<br /> | <br /> | ||
[[File:Clear improvement graph.png|Clear improvement graph]] | [[File:Clear improvement graph.png|Clear improvement graph]] | ||
<br /> | <br /> | ||
There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br /> | <br /> | ||
* There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br /> | |||
[[File:Unstable improvement graph.png|Unstable improvement graph]] | [[File:Unstable improvement graph.png|Unstable improvement graph]] | ||
<br /> | <br /> | ||
There are cases when there's a revision(s) gap between the data points in the graph, and a little investigation might save us manual bisection: | <br /> | ||
* There are cases when there's a revision(s) gap between the data points in the graph, and a little investigation might save us manual bisection: | |||
https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6f5cd956-d656-488c-bc91-b085009703c1 | https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6f5cd956-d656-488c-bc91-b085009703c1 | ||
<br /> | <br /> | ||
A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, if you haven’t already: | <br /> | ||
* A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, if you haven’t already: | |||
* Ask the other sheriffs in your team. | ** Ask the other sheriffs in your team. | ||
* See if there were situations like this in the past and find out how they were treated (there’s a high change there were). | ** See if there were situations like this in the past and find out how they were treated (there’s a high change there were). | ||
* Search for the test framework in [https://wiki.mozilla.org/Modules/All individual module ownership page] or just search for what you want in [https://wiki.mozilla.org mozilla wiki]. | ** Search for the test framework in [https://wiki.mozilla.org/Modules/All individual module ownership page] or just search for what you want in [https://wiki.mozilla.org mozilla wiki]. | ||
* Try to reach the framework owner you find in [https://wiki.mozilla.org/Modules/All individual module ownership page] by [https://chat.mozilla.org Mozilla's chat app], email or other method handy for you. | ** Try to reach the framework owner you find in [https://wiki.mozilla.org/Modules/All individual module ownership page] by [https://chat.mozilla.org Mozilla's chat app], email or other method handy for you. | ||
* [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful. | ** [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful. | ||
* If you still don’t figure it out, ask your '''team lead'''. | ** If you still don’t figure it out, ask your '''team lead'''. | ||
<br /> | |||
<br /> | |||
The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br /> | The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br /> | ||
[[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br /> | [[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br /> | ||
<br /> | <br /> | ||
A less common case of regression/improvement is when the graph is pretty clear about the culprit but the patch contains changes unrelated to the platform(s) targeted by the alert. For example, if the patch is just modifying configuration stuff for mobile platforms and the alert targets only desktop platforms, it '''might''' be an error somewhere. The '''recommendation''' is to ask in the culprit bug about what might be missing before opening the regression/linking the improvement to it. | A less common case of regression/improvement is when the graph is pretty clear about the culprit but the patch contains changes unrelated to the platform(s) targeted by the alert. For example, if the patch is just modifying configuration stuff for mobile platforms and the alert targets only desktop platforms, it '''might''' be an error somewhere. The '''recommendation''' is to ask in the culprit bug about what might be missing before opening the regression/linking the improvement to it. | ||
= Handling alerts = | = Handling alerts = | ||