Confirmed users
3,339
edits
TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions
Jump to navigation
Jump to search
m
TestEngineering/Performance/Sheriffing/Workflow (view source)
Revision as of 23:48, 23 June 2023
, 23 June 2023→How to investigate an alert: typos
(→Special Cases: Add information for sheriffing firefox-android alerts.) |
m (→How to investigate an alert: typos) |
||
| Line 45: | Line 45: | ||
* zoom in until you can easily distinguish the datapoints | * zoom in until you can easily distinguish the datapoints | ||
* retrigger more if the graph is too unstable for the data you have | * retrigger more if the graph is too unstable for the data you have | ||
* when the graph is too noisy and zooming in only makes it harder: zoom out, | * when the graph is too noisy and zooming in only makes it harder: zoom out, choose a larger timeframe (30 days, 60 days, etc), and then zoom back in slowly, following the limit line of the values in the before half of the graph until you find the datapoint that changes the trend. | ||
<br /> | <br /> | ||
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto"> | <div class="toccolours mw-collapsible mw-collapsed" style="width:auto"> | ||
| Line 87: | Line 87: | ||
There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br /> | There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br /> | ||
[[File:Unstable improvement graph.png|Unstable improvement graph]]<br /> | [[File:Unstable improvement graph.png|Unstable improvement graph]]<br /> | ||
A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, | A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, if you haven’t already: | ||
* | |||
* | * Ask the other sheriffs in your team. | ||
* | * See if there were situations like this in the past and find out how they were treated (there’s a high change there were). | ||
* | * Search for the test framework in [https://wiki.mozilla.org/Modules/All individual module ownership page] or just search for what you want in [https://wiki.mozilla.org mozilla wiki]. | ||
* [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful | * Try to reach the framework owner you find in [https://wiki.mozilla.org/Modules/All individual module ownership page] by [https://chat.mozilla.org Mozilla's chat app], email or other method handy for you. | ||
* If you still don’t figure it out, ask your '''team lead''' | * [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful. | ||
* If you still don’t figure it out, ask your '''team lead'''. | |||
The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br /> | The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br /> | ||
[[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br /> | [[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br /> | ||