Confirmed users
2,197
edits
Buildbot/Talos/Sheriffing: Difference between revisions
Redirected page to TestEngineering/Performance/Sheriffing
m (→Fix link text) |
(Redirected page to TestEngineering/Performance/Sheriffing) |
||
| (7 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
#REDIRECT [[TestEngineering/Performance/Sheriffing]] | |||
= Overview = | = Overview = | ||
The code sheriff team does a great job of finding regressions in unittests and getting fixes for them or backing stuff out. This keeps our trees green and usable while thousands of checkins a month take place! | The code sheriff team does a great job of finding regressions in unittests and getting fixes for them or backing stuff out. This keeps our trees green and usable while thousands of checkins a month take place! | ||
| Line 38: | Line 40: | ||
== Determining the root cause from Perfherder == | == Determining the root cause from Perfherder == | ||
When viewing a single alert and clicking on the graph link, Perfherder automatically show multiple branches for the given test/platform. This helps you determine the root branch. It is best to [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/ | When viewing a single alert and clicking on the graph link, Perfherder automatically show multiple branches for the given test/platform. This helps you determine the root branch. It is best to [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Perfherder_FAQ#Zooming zoom] in and out to verify where the regression is. | ||
While this isn't always clear, most of the time it is easy to see another alert on a different branch and mark the current one as a downstream if needed. | While this isn't always clear, most of the time it is easy to see another alert on a different branch and mark the current one as a downstream if needed. | ||
| Line 70: | Line 72: | ||
== Determining the scope of the regression from Perfherder == | == Determining the scope of the regression from Perfherder == | ||
Once you have the spot, you can validate the other platforms by [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/ | Once you have the spot, you can validate the other platforms by [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Perfherder_FAQ#Adding_additional_data_points adding additional data sets] to the graph. It is best here to zoom out a bit as the regression might be a few revisions off on different platforms due to [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ#What_is_coalescing coalescing]. | ||
== Cases to watch out for == | == Cases to watch out for == | ||
| Line 82: | Line 84: | ||
= Tracking bugs = | = Tracking bugs = | ||
Every release of Firefox we create a tracking bug ( | Every release of Firefox we create a tracking bug (i.e. {{bug|1122690}} - Firefox 38) which we use to associate all regressions found during that release. The reason for this is 2 fold: | ||
* We can go to one spot and see what regressions we have for reference on new bugs or to follow up. | * We can go to one spot and see what regressions we have for reference on new bugs or to follow up. | ||
* When we [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ#What_is_an_uplift uplift] it is important to see which alerts we are expecting | * When we [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ#What_is_an_uplift uplift] it is important to see which alerts we are expecting | ||
| Line 94: | Line 96: | ||
* Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos] | * Product/Component - this should be the same as the bug which is the root cause, if >1 bug, file in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos Talos] | ||
* Dependent/Block bugs - For a new bug, add the [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing#Tracking_bugs tracking bug] (for the current version) and root cause bug(s) as blocking this bug | * Dependent/Block bugs - For a new bug, add the [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing#Tracking_bugs tracking bug] (for the current version) and root cause bug(s) as blocking this bug | ||
* CC list - cc :jmaher, :avih, :wlach, : | * CC list - cc :jmaher, :avih, :wlach, :rwood, patch author(s) and reviewer(s), and owner of the tests as documented on the [https://wiki.mozilla.org/Buildbot/Talos/Tests talos tests wiki] | ||
* Summary of bug should have a check to make sure the revision is accurate | * Summary of bug should have a check to make sure the revision is accurate | ||
* The description is auto suggested as well, please verify the revision here | * The description is auto suggested as well, please verify the revision here | ||
| Line 109: | Line 111: | ||
== Merge Day - Uplifts == | == Merge Day - Uplifts == | ||
Every 6 weeks we do an [https://wiki.mozilla.org/Buildbot/Talos/ | Every 6 weeks we do an [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ#What_is_an_uplift uplift]. These typically result in [https://elvis314.wordpress.com/2014/12/12/tracking-firefox-performance-as-we-uplift-the-volume-of-alerts-we-get/ dozens of alerts] for each uplift. | ||
The job here is to triage alerts as we usually do, except in this case we have a much larger volume of alerts. One thing here is we have alerts from the upstream branch. Take for example when we uplift Mozilla-Central to Mozilla-Aurora. We have a tracking bug for each release, and there is a list of bugs (keep in mind some are resolved as wontfix). In a perfect world (half the time) we can match up the alerts that are showing up on Mozilla-Aurora with the bugs that have already been filed. The job here is to verify and add bugs to keep track of what is there. | The job here is to triage alerts as we usually do, except in this case we have a much larger volume of alerts. One thing here is we have alerts from the upstream branch. Take for example when we uplift Mozilla-Central to Mozilla-Aurora. We have a tracking bug for each release, and there is a list of bugs (keep in mind some are resolved as wontfix). In a perfect world (half the time) we can match up the alerts that are showing up on Mozilla-Aurora with the bugs that have already been filed. The job here is to verify and add bugs to keep track of what is there. | ||
| Line 123: | Line 125: | ||
* [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Alert_FAQ Alert FAQ] | * [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Alert_FAQ Alert FAQ] | ||
* [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Noise_FAQ Noise FAQ] | * [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Noise_FAQ Noise FAQ] | ||
* [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/ | * [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Perfherder_FAQ Perfherder FAQ] | ||
* [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ Tree FAQ] | * [https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Tree_FAQ Tree FAQ] | ||
* [https://wiki.mozilla.org/Performance_sheriffing go to new page] | |||