Sheriffing/Schedule for Tasks performed by Code Sheriffs: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(added initial list of tasks)
 
(update tasks Aryx)
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
Version: 1 (Last Updated 2018-08-21)
Version: 8 (Last Updated 2021-08-10)
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 6: Line 6:
| Watch trees|| Always
| Watch trees|| Always
||Watch the following trees for failures, classify them, do backouts and escalate infra issues:
||Watch the following trees for failures, classify them, do backouts and escalate infra issues:
* trunk (mozilla-central, autoland, mozilla-inbound)  
* trunk (mozilla-central, autoland)  
* mozilla-beta  
* mozilla-beta  
* mozilla-release  
* mozilla-release  
Line 18: Line 18:
||Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
||Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
|- style="vertical-align:top"
|- style="vertical-align:top"
| Land patches and do uplifts to beta || Every 2h
| Merge autoland tree to mozilla-central || Mo-Fr every 6h
||Check for patches waiting to land (have keyword “checkin-needed” at Bugzilla) or to be uplifted from central to beta. Beta patches shall have landed and build and test coverage before 9am Pacific Time on Mondays and Thursdays when betas are built. Release Management might request to land more on short notice.
|- style="vertical-align:top"
| Merge integration trees || Mo-Fr every 6h
Sa+Su: every 12h
Sa+Su: every 12h
|| Merge autoland and mozilla-inbound to mozilla-central and back
|| Merge autoland to mozilla-central and mozilla-central to autoland if necessary
|- style="vertical-align:top"
|- style="vertical-align:top"
| Beta simulations || Daily
| central-as-beta simulations || Early Beta: every day
Late Beta: Tue+Fri
||Simulate the code in mozilla-central as mozilla-beta
||Simulate the code in mozilla-central as mozilla-beta
|- style="vertical-align:top"
|- style="vertical-align:top"
| Merge mozilla-central to mozilla-beta || Daily if late in release cycle
| beta-as-release simulation || Thursdays
||After beta simulations without any existing perma failures, merge mozilla-central to mozilla-beta. Use the revisions from the beta simulations.
||Simulate the code in mozilla-beta as mozilla-release, skip in last week of development cycle when beta already got merged to release
See dates in beta simulation document when this has to be done.
|- style="vertical-align:top"
| Intermittent triage || Weekend
||Frequent failure escalation or reminders to developers
|- style="vertical-align:top"
|- style="vertical-align:top"
| Clean up list of bugs which don’t get suggested || Weekly
| Clean up list of bugs which don’t get suggested || Weekly
Line 37: Line 37:
# If the bug has been resolved as incomplete, remove it from the document.
# If the bug has been resolved as incomplete, remove it from the document.
# If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.
# If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.
|}
====Release graphs====
In the nights from
* Sunday to Monday
* Tuesday to Wednesday
* Thursday to Friday
the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).
These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).
Exceptions are:
* Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
* week before release: release candidates get built instead and release management starts these manually.
Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.
* Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty] channel on Matrix if they continue to fail.
* 5h after the release graphs got scheduled and posted in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty channel] (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
** The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
** Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty channel] and create a bug if there is no reply in 15 minutes.
*** These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
*** Needinfo the triage owner and the [[Release_Management/Release_owners|release manager who owns the release]].
** Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
** The same process also applies to "rapid" releases and "ESR" releases.
====Sheriffing and quality/stability related tasks executed by one person (Aryx)====
{| class="wikitable"
|-
! Task !! Schedule !! Details
|- style="vertical-align:top"
| version increase simulation || Second Tuesday after version increase || The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let [https://github.com/mozilla/probe-scraper probe-scraper] alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.
Simulate the code in mozilla-central with the next higher version number.
|- style="vertical-align:top"
| Create new beta simulation document|| Monday on which version got increased
||Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management.
|- style="vertical-align:top"
| Add new version to tree statistics || Monday on which version got increased
||Tool which shows how often and how long the trees got closed
* [https://github.com/Archaeopteryx/treestats code]
* [https://archaeopteryx.github.io/treestats/ tool]
|- style="vertical-align:top"
| Increase simulation version numbers || Monday on which version got increased
||Increment version numbers for central-as-beta and beta-as-release simulations at [[Sheriffing/How To/Beta simulations]]
|- style="vertical-align:top"
| Land patches || daily
||Check for patches waiting to land
|}
|}

Latest revision as of 14:02, 30 September 2024

Version: 8 (Last Updated 2021-08-10)

Task Schedule Details
Watch trees Always Watch the following trees for failures, classify them, do backouts and escalate infra issues:
  • trunk (mozilla-central, autoland)
  • mozilla-beta
  • mozilla-release
  • mozilla-esr trees (all of them which are available in Treeherder; often there is only 1, but for some time 2 when their cycles overlap)

Open them up to the oldest push with unclassified failures (excluding backfills and retriggers added >4h after the push) or running jobs.

Email reading Start of shift, every 30 minutes Help other people by answering needinfo requests etc.
Read new, not suggested bugs Start of shift Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
Merge autoland tree to mozilla-central Mo-Fr every 6h

Sa+Su: every 12h

Merge autoland to mozilla-central and mozilla-central to autoland if necessary
central-as-beta simulations Early Beta: every day

Late Beta: Tue+Fri

Simulate the code in mozilla-central as mozilla-beta
beta-as-release simulation Thursdays Simulate the code in mozilla-beta as mozilla-release, skip in last week of development cycle when beta already got merged to release
Intermittent triage Weekend Frequent failure escalation or reminders to developers
Clean up list of bugs which don’t get suggested Weekly Check if bugs are still active or needed for beta. Preferable on the weekend when there is more time for this.
  1. Open the Bugzilla links from the document.
  2. If the bug has been resolved as incomplete, remove it from the document.
  3. If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.

Release graphs

In the nights from

  • Sunday to Monday
  • Tuesday to Wednesday
  • Thursday to Friday

the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).

These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).

Exceptions are:

  • Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
  • week before release: release candidates get built instead and release management starts these manually.

Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.

  • Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the releaseduty channel on Matrix if they continue to fail.
  • 5h after the release graphs got scheduled and posted in the releaseduty channel (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
    • The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
    • Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the releaseduty channel and create a bug if there is no reply in 15 minutes.
      • These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
      • Needinfo the triage owner and the release manager who owns the release.
    • Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
    • The same process also applies to "rapid" releases and "ESR" releases.

Sheriffing and quality/stability related tasks executed by one person (Aryx)

Task Schedule Details
version increase simulation Second Tuesday after version increase The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let probe-scraper alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.

Simulate the code in mozilla-central with the next higher version number.

Create new beta simulation document Monday on which version got increased Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management.
Add new version to tree statistics Monday on which version got increased Tool which shows how often and how long the trees got closed
Increase simulation version numbers Monday on which version got increased Increment version numbers for central-as-beta and beta-as-release simulations at Sheriffing/How To/Beta simulations
Land patches daily Check for patches waiting to land