MozillaWiki

TestEngineering/Performance/Sheriffing/Workflow: Difference between revisions

Page Discussion

TestEngineering/Performance/Sheriffing/Workflow (view source)

Revision as of 12:33, 10 July 2024

13,804 bytes added ,  10 July 2024
→‎Alert caused by a backout: removed section because we added the details elsewhere
(→‎Alert caused by a backout: removed section because we added the details elsewhere)
 
(37 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= Context =
= Context =
Performance sheriffs, along with the standard [[TestEngineering/Performance/Sheriffing#Overview|by-the-book definition]], are responsible for checking for performance regressions on a daily basis. This is done by reviewing alerts from our performance tests on [https://treeherder.mozilla.org/perf.html Perfherder]. Any time a test or tests exceed [[TestEngineering/Performance/Sheriffing/Workflow#Thresholds|the set threshold for its framework]], one or multiple alerts will be generated. The goal of the sheriff is to identify the commit or revision responsible for the regression, and file a bug with a :needinfo flag for the author(s) of that commit or revision.
Performance sheriffs, along with the standard [[TestEngineering/Performance/Sheriffing#Overview|by-the-book definition]], are responsible for checking for performance regressions on a daily basis. This is done by reviewing alerts from our performance tests on [https://treeherder.mozilla.org/perf.html Perfherder]. Any time a test or tests exceed [[TestEngineering/Performance/Sheriffing/Workflow#Thresholds|the set threshold for its framework]], one or multiple alerts will be generated. The goal of the sheriff is to identify the commit or revision responsible for the regression, and file a bug with a :needinfo flag for the author(s) of that commit or revision.
= How to Become a Sheriff =
To become a sheriff(and have sheriffing permissions) you will need to file 2 bugs to have sheriffing actions allowed in treeherder.
1) Request to be added to the treeherder sheriff group
- Add your name in the title and in the description add your ldap email address in the
[https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=nobody%40mozilla.org&bug_ignored=0&bug_severity=--&bug_status=NEW&bug_type=task&cf_a11y_review_project_flag=---&cf_accessibility_severity=---&cf_fx_iteration=---&cf_fx_points=---&comment=I%20will%20be%20sheriffing%20performance%20alerts%20on%20Perfherder%2C%20so%20I%20would%20like%20to%20be%20added%20to%20the%20Treeherder%20sheriff%20group.%0D%0A%0D%0AMy%20LDAP%20address%20is%3A%20&component=Treeherder%3A%20Infrastructure&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_cc=cpeterson%40mozilla.com%2C%20cvalaas%40mozilla.com%2C%20dave.hunt%40gmail.com%2C%20jdescottes%40mozilla.com%2C%20sclements%40mozilla.com&defined_groups=1&filed_via=standard_form&flag_type-4=X&flag_type-41=X&flag_type-607=X&flag_type-803=X&flag_type-936=X&needinfo_role=other&needinfo_type=needinfo_from&op_sys=Unspecified&priority=--&product=Tree%20Management&rep_platform=Unspecified&short_desc=Add%20%3Cname%3E%20to%20Treeherder%20sheriff%20group&target_milestone=---&version=--- linked bug]
2) Add yourself to the perf_sheriff group
- Add name and ldap email in title and email in the description of the
[https://bugzilla.mozilla.org/enter_bug.cgi?assigned_to=infra%40infra-ops.bugs&bug_ignored=0&bug_severity=--&bug_status=NEW&bug_type=task&cf_fx_iteration=---&cf_fx_points=---&comment=I%20will%20be%20sheriffing%20performance%20alerts%20on%20Perfherder%2C%20so%20I%20would%20like%20to%20be%20added%20to%20the%20Treeherder%20sheriff%20group.%0D%0A%0D%0AMy%20LDAP%20address%20is%3A%20&component=Infrastructure%3A%20LDAP&contenttypemethod=list&contenttypeselection=text%2Fplain&defined_cc=cpeterson%40mozilla.com%2C%20&defined_groups=1&filed_via=standard_form&flag_type-4=X&flag_type-607=X&flag_type-674=X&flag_type-803=X&flag_type-936=X&groups=mozilla-employee-confidential&needinfo_role=other&needinfo_type=needinfo_from&op_sys=Unspecified&priority=--&product=Infrastructure%20%26%20Operations&rep_platform=Unspecified&short_desc=Add%20me%28%3CLDAP%20EMAIL%3E%29%20to%20perf_sheriff%20LDAP%20group&target_milestone=---&version=unspecified linked bug]
Once both bugs are completed check that you can do sheriffing actions. You should be able to now update alerts through the ability to add notes and tags to alerts.


= Flowchart =
= Flowchart =
Line 41: Line 56:
[[File:Alert assigned to - small.png|Alert assigned to - small]].<br />
[[File:Alert assigned to - small.png|Alert assigned to - small]].<br />
<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Golden rules ==
== Reading the graph ==
    <div class="mw-collapsible-content">
As '''golden rules''', when you are not sure about the culprit of the graph you're investigating:
As '''golden rules''', when you are not sure about the culprit of the graph you're investigating:
* zoom in until you can easily distinguish the datapoints
* zoom in until you can easily distinguish the datapoints
* retrigger more if the graph is too unstable for the data you have
* retrigger more if the graph is too unstable for the data you have
* when the graph is too noisy and zooming in only makes it harder: zoom out, chose a larger timeframe (30 days, 60 days, etc)
* when the graph is too noisy and zooming in only makes it harder: zoom out, choose a larger timeframe (30 days, 60 days, etc), and then zoom back in slowly, following the limit line of the values in the before half of the graph until you find the datapoint that changes the trend.
<br />
<br />
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Reading the graph ==
    <div class="mw-collapsible-content">
<br />
<br />
To read the graph of a certain alert, you just need to put the mouse over it and click on the '''graph''' link that appears:
To read the graph of a certain alert, you just need to put the mouse over it and click on the '''graph''' link that appears:
Line 67: Line 83:
<br />
<br />
TODO: list categories of graph - stable, noisy (noisy with bigger or smaller trend change), modal, invalid.
TODO: list categories of graph - stable, noisy (noisy with bigger or smaller trend change), modal, invalid.
</div>
</div>
== Retrigger/Backfill ==
== Retrigger/Backfill ==
Depending on the test, the jobs are ran once in several revisions or on every revision. The vast majority runs once in several revision, so almost every time you need to do a backfill between the first bad and last good job in order to determine for sure the culprit, by clicking on the '''job''' link from the tooltip of the regression data-point.
Depending on the test, the jobs are ran once in several revisions or on every revision. The vast majority runs once in several revision, so almost every time you need to do a backfill between the first bad and last good job in order to determine for sure the culprit, by clicking on the '''job''' link from the tooltip of the regression data-point.
Line 80: Line 99:
After the jobs are finished running, you have jobs triggered on successive revisions and you can continue looking for the culprit.<br />
After the jobs are finished running, you have jobs triggered on successive revisions and you can continue looking for the culprit.<br />
== Finding the culprit ==
== Finding the culprit ==
A clear improvement/regression appears usually when there is easily noticeable difference between two adjacent data points:
* A clear improvement/regression appears usually when there is easily noticeable difference between two adjacent data points:
<br />
<br />
[[File:Clear improvement graph.png|Clear improvement graph]]
[[File:Clear improvement graph.png|Clear improvement graph]]
<br />
<br />
There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br />
<br />
[[File:Unstable improvement graph.png|Unstable improvement graph]]<br />
* There are cases when the difference is much less noticeable and the data of the test is more unstable, and some retriggers are necessary in order to determine the interval for the test data and compare it between several adjacent tests:<br />
A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, it you haven’t already:
[[File:Unstable improvement graph.png|Unstable improvement graph]]
* from the other sheriffs in your team
<br />
* see if there were situations like this in the past and find out how they were treated (there’s a high change there were)
<br />
* search for the test framework in [https://wiki.mozilla.org/Modules/All individual module ownership page] or just search for what you want in [https://wiki.mozilla.org mozilla wiki]
* There are cases when there's a revision(s) gap between the data points in the graph, and a little investigation might save us manual bisection:
* try to reach the framework owner you find in [https://wiki.mozilla.org/Modules/All individual module ownership page] by [https://chat.mozilla.or mozilla's chat app], email or other method handy for you
https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6f5cd956-d656-488c-bc91-b085009703c1
* [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful
<br />
* If you still don’t figure it out, ask your '''team lead'''
<br />
* A less fortunate situation is when the test is unstable, there are gaps in graph when the tests didn’t run for various reasons and the regression/improvement is almost impossible to be determined. If the investigation takes more than 5 business days it’s recommended to ask for help, if you haven’t already:
** Ask the other sheriffs in your team.
** See if there were situations like this in the past and find out how they were treated (there’s a high change there were).
** Search for the test framework in [https://wiki.mozilla.org/Modules/All individual module ownership page] or just search for what you want in [https://wiki.mozilla.org mozilla wiki].
** Try to reach the framework owner you find in [https://wiki.mozilla.org/Modules/All individual module ownership page] by [https://chat.mozilla.org Mozilla's chat app], email or other method handy for you.
** [https://sso.mozilla.com/dashboard Mozilla dashboard] might also be helpful.
** If you still don’t figure it out, ask your '''team lead'''.
<br />
<br />
The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br />
The investigation might end up opening a bug without knowing the specific commit that caused the regression and asking for help from most relevant people you found about. <br />
[[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br />
[[File:Graph uncertain culprit.png|1200px|Graph uncertain culprit]]<br />
<br />
<br />
A less common case of regression/improvement is when the graph is pretty clear about the culprit but the patch contains changes unrelated to the platform(s) targeted by the alert. For example, if the patch is just modifying configuration stuff for mobile platforms and the alert targets only desktop platforms, it '''might''' be an error somewhere. The '''recommendation''' is to ask in the culprit bug about what might be missing before opening the regression/linking the improvement to it.
A less common case of regression/improvement is when the graph is pretty clear about the culprit but the patch contains changes unrelated to the platform(s) targeted by the alert. For example, if the patch is just modifying configuration stuff for mobile platforms and the alert targets only desktop platforms, it '''might''' be an error somewhere. The '''recommendation''' is to ask in the culprit bug about what might be missing before opening the regression/linking the improvement to it.
    </div>
</div>


= Handling alerts =
= Handling alerts =
When an alert is created, it is not necessarily caused by a bug. It can be also caused by the instability/noise of the test or by other causes that are unrelated to the repo code, like CI setup.
When an alert is created, it is not necessarily caused by a bug. It can be also caused by the instability/noise of the test or by other causes that are unrelated to the repo code, like CI setup.
<br />
<br />
<br />
A bug associated with an alert needs to have the perf-alert keyword. The regression bugs have that automatically when the File bug button is pressed, but for the bugs that are not regression bugs needs to be added manually.
== Thresholds ==
== Thresholds ==
There are different thresholds above which the alerts are considered regressions and they vary depending on the framework:
There are different thresholds above which the alerts are considered regressions and they vary depending on the framework:
Line 115: Line 143:
== Identifying the culprit ==
== Identifying the culprit ==
If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here:
If the revision that caused the regression is clear, then what is left to do is identify the culprit bug. There are several situations here:
* The revision contains changes only from one bug and you need to open a regression bug for that
 
* The revision contains changes from several bugs but you are familiar with the test and you know which of the bugs caused the regression and open a bug for that
* If the revision contains changes only from one bug, you need to open a regression bug for that.
* The revision contains changes from several bugs (usually a merge from one of the other repos) when you need to do a [https://docs.google.com/document/d/1SP3rckeNOuZ6X-LzSEPjoGqQICApIzIiPlgrgJmdU38 bisection] in order to identify the causing bug.
* If the revision contains changes from several bugs but you are familiar with the test and you know which of the bugs caused the regression, open a bug for that one.
* If the revision contains changes from several bugs (usually a merge from one of the other repos), you need to do a [https://docs.google.com/document/d/1SP3rckeNOuZ6X-LzSEPjoGqQICApIzIiPlgrgJmdU38 bisection] in order to identify the causing bug.


=== Harness alerts ===
=== Harness alerts ===
Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as wontfix and add the "harness" tag. Otherwise, open regression bug.
Harness alerts are usually caused by re-recordings or changes to code from testing/raptor component changing the baseline. If the regression is assumed, then link the culprit directly to the alert, close as WONTFIX and add the "harness" tag. Otherwise, open a regression bug.
 
===== Baseline Changes =====
 
Baseline changes are hard to identify, but we have some questions we can ask ourselves and with other people on the team to help determine if we are dealing with a real regression or a baseline change:
 
1) Should the patch that caused a change in the baseline metrics because we are aiming for testing in a different environment?
 
If yes, then the new baseline may be accepted. As the conditions for testing have changed we have to expect that some things will change. Take for instance adding a blank profile to the conditioned profile changes, as this is testing in a different environment we expect things will change
 
2) Is the patch changing something that shouldn't cause a shift in metrics?
 
If yes, then do not accept the new baseline immediately, do an analysis and discuss with other sheriffs to get second opinions.
 
3) Does the patch change the test itself?
 
If yes, a new baseline may be accepted depending on the changes. If the changes are not functional, then we shouldn’t accept a new baseline.
 
4) Are test recordings changing?
 
If yes, then a new baseline should be accepted after checking to make sure that the page that is being tested is different from the previous recording. If they aren’t, then the differences should be investigated further - it’s possible that they still may be a valid baseline change even if the page is the same, as non-visual changes may be different.
 
5) Any other cases that do not fall in these categories should be discussed with the team for second opinions and with patch authors if needed.


=== Backout/regression-fix alerts ===
=== Backout/regression-fix alerts ===
These alerts are caused by backouts or fixes of regressions and the associated tags are regression-backedout respectively regression-fix. If they are regression, they get WONTFIX status.  
These alerts are caused by backouts or fixes of regressions and the associated tags are regression-backedout respectively regression-fix. The status of regressions can be changed to WONTFIX.  
 
For alerts caused by backout the process is to reply to the backout comment with the summary of the alert and adding the '''''perf-alert''''' keyword. 
<br/>
<br/>


=== Infra alerts ===
=== Infra alerts ===
The infra regressions are caused by infra changes are probably the most difficult to identify. Excepting the case when the infra change is announced and known of, usually an infra regression is most likely to be detected by the sheriff after all the suspect commits/bugs were removed from the list.
 
==== What is an infra alert ====
 
The regressions caused by infra changes are probably the most difficult to identify. Excepting the case when the infra change is announced and known of, an infra regression is usually most likely to be detected by the sheriff after all the suspect commits/bugs were removed from the list.
<br />
<br />
<br />
<br />
Line 133: Line 189:
<br />
<br />
<br />
<br />
Looking at the graph below, it is obvious that the data-points highlighted around '''Sep 10''' vary in the same interval as the data-points after '''Sep 13'''. They were retriggered after this date, when the infra change happened. The datapoint around '''Sep 10''' that are not highlighted were triggered before the infra change happened, and you can see that their vary interval is constant lower.<br />
Looking at the graph below, it is obvious that until '''Apr 17''' the graph is constantly around 1800-2000 and after that it dropped to 1200-1400. The backfill didn't reveal the culprit so we retriggered a while back. The retriggers were in the interval of the improvement and we can also see that there are many yellow vertical lines that mark the infra changes.<br />
[[File:Grapg infra regression.png|500px|Graph infra regression]]<br />
[[File:Grapg infra change.png|1200px|Graph infra change]]<br />
<br />
For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes related to the infrastructure that is very useful when the investigation is leading to this kind of regression: https://changelog.dev.mozaws.net/
<br />
<br />
==== How to handle infra alerts ====
Infra alerts became more frequent starting sometime in 2022, but the causes are still unknown for the moment. They usually come up in pairs of regression-improvement. When a regression comes up:
<br />
* it has to be tagged with the '''infra''' tag and move it to '''investigating''' queue
As soon as it comes back, if it is an improvement:
<br />
* it has to be tagged with the '''infra, regression-fix, improvement''' tags and mark it as WONTFIX
<br />
'''OR''' if the alert contains regression alerts:
<br />
<br />
For an easy follow-up, there’s a [https://changelog.dev.mozaws.net/ changelog] containing the changes realted to the infrastructure that is very useful when the investigation is leading to this kind of regression: https://changelog.dev.mozaws.net/
* it has to be tagged with the '''infra, regression-fix'''' tags and mark it as WONTFIX
<br />
as well as the regression alert
<br />
* has to be changed the status to '''FIXED'''


=== Invalid alerts ===
=== Invalid alerts ===
Line 142: Line 215:
<br />
<br />
<br />
<br />
In the graph below, the regression appeared around '''Dec 9''' and as you can see, there a pattern of vary predominantly between 0.7 and 1. If you click on first highlighted datapoint (around '''Dec 3''') you’ll see that its alert is marked as invalid.
In the graph below, the regression appeared around '''Dec 9''' and as you can see, there a pattern of vary predominantly between 0.7 and 1. If you click on first highlighted datapoint (around '''Dec 3'''), you’ll see that its alert is marked as invalid.
<br />
<br />
<br />
<br />
Line 151: Line 224:
[[File:Graph sccache real regression.png|Graph sccache real regression]]<br />
[[File:Graph sccache real regression.png|Graph sccache real regression]]<br />
<br />
<br />
A particular case are '''sccache hit rate''' tests. Most often, those alerts are invalid, but is the hit rate drops stay stays low for at least '''12-24h''' then a regression bug should be open.
A particular case are '''sccache hit rate''' tests. Most often, those alerts are invalid, but is the hit rate drops stay stays low for at least '''12-24h''' then a regression bug should be open.<br />
 
<br />
== Handling Regressions ==
== Handling Regressions ==
There are two different approaches to handling regressions:
There are two different approaches to handling regressions:
Line 222: Line 295:


== Handling improvements ==
== Handling improvements ==
Unlike for regressions, when you identified an improvement there's no need to open a bug, you just need to notify the bug assignee via a comment and add the 'perf-alert' keyword.
 
Unlike for regressions, when you have identified an improvement, there's no need to open a bug. Just notify the bug assignee via a comment and add the 'perf-alert' keyword.


=== Valid improvements ===
=== Valid improvements ===
Line 235: Line 309:


You also need to add the 'perf-alert' keyword, to indicate there is a performance alert associated with this bug.
You also need to add the 'perf-alert' keyword, to indicate there is a performance alert associated with this bug.
Note: If by mistake you left a comment that is actually invalid, just add to the comment the tag '''obsolete'''.


'''Attention!''' The summary could contain alerts reassigned from other alerts. You have to tick the box next to '''each untriaged alert''' and change its status to ''Acknowledge''. <br />
'''Attention!''' The summary could contain alerts reassigned from other alerts. You have to tick the box next to '''each untriaged alert''' and change its status to ''Acknowledge''. <br />
Line 257: Line 333:
Update the notes tag if it fits to one of the following:
Update the notes tag if it fits to one of the following:
=== Improvements ===
=== Improvements ===
* harness - patches that updated that harness and caused improvements
* harness - patches that updated that harness and caused improvements
* regression-backedout - paches backed out due to causing regressions  
* regression-backedout - patches backed out due to causing regressions  
* infra - improvements caused by infra changes (cheanges not related to repository code)
** ''applies to the alerts linked to bugs that backed out a regression'' '''NOT''' ''to alerts that are linked to regression bugs''
* regression-fix - pacthes fixing a reported regression bug
* infra - improvements caused by infra changes (changes not related to repository code)
* improvements - pacthes causing improvements
* regression-fix - patches fixing a reported regression bug
** ''applies to the alerts linked to bugs that fixed a regression'' '''NOT''' ''to alerts that are linked to regression bugs''
* improvements - patches causing improvements
 
=== Regressions ===
=== Regressions ===
* harness - patches that updated that harness and caused regressions
* harness - patches that updated that harness and caused regressions
* infra - regressions caused by infra changes (cheanges not related to repository code)
* regression-backedout - patches backed out due to causing regressions
** ''applies to the alerts linked to bugs that backed out a regression'' '''NOT''' ''to alerts that are linked to regression bugs''
* infra - regressions caused by infra changes (changes not related to repository code)
* regression-fix - patches fixing a reported regression bug
** ''applies to the alerts linked to bugs that fixed a regression'' '''NOT''' ''to alerts that are linked to regression bugs''
* improvement-backedout - regression caused by backing out an earlier improvement


== Move alerts from untriaged queue ==
== Move alerts from untriaged queue ==
Line 292: Line 378:
= Special Cases =
= Special Cases =
== Multiple Bug IDs on the same push ==
== Multiple Bug IDs on the same push ==
=== Regressions ===
If the push that has been identified as the culprit has multiple revisions and bug ids linked to it, the sheriff can fill in the "Regressed By" field for each of the bugs with the value of the filed bug for the associated alert. <br/>
If the push that has been identified as the culprit has multiple revisions and bug ids linked to it, the sheriff can fill in the "Regressed By" field for each of the bugs with the value of the filed bug for the associated alert. <br/>
TODO: add screenshot for example [https://treeherder.mozilla.org/jobs?repo=autoland&revision=88a481adf83070d1cdcc2bb074f6545b5dfe97e7&group_state=expanded&selectedTaskRun=GAP1DzYETPawhJhP1WkzGw.0/ alert]
If there's a regression-fix, regression-backedout or improvement-backedout bug to be linked to the alert but the revision contains multiple bugs, link the most relevant bug to the alert. If you're unsure, ni? the author to tell which of the bugs they think caused the alert.
 
=== Improvements ===
The general approach for improvements is to leave needinfos to the authors to ask for help in identifiying the corresponding bug.
 
== Sheriffing Firefox-Android Alerts ==
The [https://treeherder.mozilla.org/jobs?repo=firefox-android| firefox-android repo] (which contains both Fenix and Focus) also produces alerts at a lower cadence (~6 days after a change is triggered).
 
Sheriffing this branch is the same as autoland in that you'll need to do backfills on the firefox-android branch to determine a culprit commit. Once you find a culprit commit, then file a bug in Bugzilla as usual, and needinfo the author of the commit.
 
If you find a commit that has no clear author [https://treeherder.mozilla.org/jobs?repo=firefox-android&revision=96420c198887c5148a2a5b80b947c6891403d0f2| such as this one], then you'll have one of two options depending on the type of commit it is:
 
* If the commit message mentions '''GeckoView''' or something related to mozilla-central, then needinfo Android engineering manager '''Bryan Clark [:bclark]''' in the bug for the alert.
* If the commit message doesn't mention '''GeckoView''' and has no clear author, [https://treeherder.mozilla.org/jobs?repo=firefox-android&revision=60eba6f35766d898964c409eeeacd7573812c364| like this commit], then needinfo Android engineering manager '''Joe Mahon [:jmahon]''' in the bug for the alert.
 
Check to see if there might be an associated GeckoView alert that was produced on autoland when there's an alert produced on firefox-android. If there was, then you can associated the two alert bugs with a '''See Also''' entry.
 
Lastly, backfilling on the mozilla-central side when a change is detected in a GeckoView commit is done with try runs. [https://wiki.mozilla.org/Performance/Fenix/Performance_reviews#Backfilling_to_determine_culprit_commit| You can find instructions for that here].
 
=== How to triage a firefox-android alert ===
1. Read the graph, check for job gaps (browsertime & build_metrics):
* click on the graph icon next to an alert item (usually the highest magnitude)
a thin vertical line will show for all the alerts associated with the test, you need to make sure you’re looking at the right one by hovering or * clicking on the datapoint
* click on the job link to go to treeherder job view
* make sure you’re seeing the jobs from current revision back and look for gaps
2. Trigger the jobs in the revisions in the gap (browsertime & build_metrics)
* For every revision in the gap click '''''Add new jobs''''' and trigger the needed job manually
3. Find the responsible push that caused the regression
* If the regression is caused by a mozilla-central push, then you have to take the build id (Update GeckoView (Nightly) to 121.0.'''20231106094018'''.) and search for the revision in [https://buildhub.moz.tools/ Buildhub]
* Build-metrics: file a generic performance regression
* Browsertime: it should be an autoland alert to mark as downstream for
4. If it’s caused by a firefox-android push then the process of filling a regression bug should be straight-forward - the default one from [[TestEngineering/Performance/Sheriffing/Workflow#Filing_a_regression_bug|documentation]]. (browsertime & build_metrics)
 
==== Special case ====
There’s a special case that happens very very rare when code from geckoview build (mozilla-central) doesn’t affect geckoview pageload but affects fenix (firefox-android). It is the case for alert [https://treeherder.mozilla.org/perfherder/alerts?id=40028&hideDwnToInv=0 40028]. In this case there’s no other option than bisection on mozilla-central with fenix build. The steps are:
# Follow bisection workflow from documentation
# Checkout to the previous commit of the suspect one
# Download the [https://treeherder.mozilla.org/jobs?repo=firefox-android&group_state=expanded&tochange=71d772f64a8acfd1f916d8cc7fde01957c5ca2d5&fromchange=71d772f64a8acfd1f916d8cc7fde01957c5ca2d5&searchStr=fenix-android-all%2Copt%2CNightly-related%2Ctasks%2Cthat%2Crun%2Con%2Ceach%2Cgithub%2Cpush%2Cbuild-apk-fenix-nightly-simulation%2CB&selectedTaskRun=Zxf1CST0Ru61vzqrElJp3A.0 fenix-android-all fenix-nightlySim(B) build]: [https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Zxf1CST0Ru61vzqrElJp3A/runs/0/artifacts/public/build/target.armeabi-v7a.apk target.armeabi-v7a.apk] (artifacts)
# Add it to the code using the command: ''./mach try perf --browsertime-upload-apk path/to//target.armeabi-v7a.apk''
# Commit ''hg commit -m="fenix apk"''
# Push to try the pageload job for geckoview ''Android 11.0 Samsung A51 AArch64 Shippable''
# Repeat steps 1-6 for pushing to try the suspected revision and then compare the results to try.
 
== Alerts caused by modifying release flags (EARLY_BETA_OR_EARLIER) / bumping the Firefox version ==
These alerts are generated on mozilla-beta and have no bugs associated with the revisions:
# Check the milestone version by looking into the changeset details.
# Check if there was already a bugzilla bug created for this particualr milestone version.
# If not, then clone the following bug https://bugzilla.mozilla.org/show_bug.cgi?id=1879080. <br/>
# Leave a comment with the alert summary you were investigating.
Comment 0 could include the following information: <br/>
''There were some alerts generated on mozilla-beta, that have no bugs associated with the revisions. <br/>''
''Alert: https://treeherder.mozilla.org/perf.html#/alerts?id=<alert#number> <br/>''
''Pushlog: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=<rev>&tochange=<rev> <br/>''
''Changeset Details: https://hg.mozilla.org/releases/mozilla-beta/rev/<rev> <br/>''
''A few notes to consider:''
*''for mozilla-beta alerts, we normally search for downstreams on autoland''
*''most of the time, when a change is pushed to autoland, it is also pushed to mozilla-central during the same day''
*''on mozilla-beta the change is merged after ~ 6 weeks''
 
== build_metrics special cases ==
# '''''decision''''' alerts usually are just spikes and not relevant, which is why we are setting them as ''invalid''. In the case that, by looking at the coresponding graph, the regression wasn't improved / fixed, then check with the other sheriffs in the Sheriffs of Performance channel on Element since it might be a cause of concern.
# '''''instrumented''''' in the Tags & Options column should be set as ''invalid''. Alerting will be deactivated for these tests in the future.
 
== Regression which was already backedout ==
If we identify a regression alert summary and that particular changeset was already backout then the safe approach would be to file a bug by following the usual workflow, but mention in the comment0 that this was already backedout and there's no action item for the author. This bug is only created for tracking purposes / being able to acknowledge the change summary of the alert.
 
== Jobs are failing to run ==
In this case we usually resort to bisection.
 
== Invalid comments ==
If by mistake you left a comment that is actually invalid, just add to the comment the tag '''''obsolete'''''.
 
= Resources =
= Resources =
* [https://wiki.mozilla.org/TestEngineering/Performance/FAQ FAQ]
* [https://wiki.mozilla.org/TestEngineering/Performance/FAQ FAQ]
Bacasandrei
15

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/1241645...1251275"