Sheriffing/Schedule for Tasks performed by Code Sheriffs
From MozillaWiki
Version: 8 (Last Updated 2021-08-10)
| Task | Schedule | Details |
|---|---|---|
| Watch trees | Always | Watch the following trees for failures, classify them, do backouts and escalate infra issues:
Open them up to the oldest push with unclassified failures (excluding backfills and retriggers added >4h after the push) or running jobs. |
| Email reading | Start of shift, every 30 minutes | Help other people by answering needinfo requests etc. |
| Read new, not suggested bugs | Start of shift | Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down. |
| Merge autoland tree to mozilla-central | Mo-Fr every 6h
Sa+Su: every 12h |
Merge autoland to mozilla-central and mozilla-central to autoland if necessary |
| central-as-beta simulations | Early Beta: every day
Late Beta: Tue+Fri |
Simulate the code in mozilla-central as mozilla-beta |
| beta-as-release simulation | Thursdays | Simulate the code in mozilla-beta as mozilla-release, skip in last week of development cycle when beta already got merged to release |
| Intermittent triage | Weekend | Frequent failure escalation or reminders to developers |
| Clean up list of bugs which don’t get suggested | Weekly | Check if bugs are still active or needed for beta. Preferable on the weekend when there is more time for this.
|
Release graphs
In the nights from
- Sunday to Monday
- Tuesday to Wednesday
- Thursday to Friday
the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).
These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).
Exceptions are:
- Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
- week before release: release candidates get built instead and release management starts these manually.
Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.
- Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the releaseduty channel on Matrix if they continue to fail.
- 5h after the release graphs got scheduled and posted in the releaseduty channel (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
- The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
- Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the releaseduty channel and create a bug if there is no reply in 15 minutes.
- These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
- Needinfo the triage owner and the release manager who owns the release.
- Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
- The same process also applies to "rapid" releases and "ESR" releases.
| Task | Schedule | Details |
|---|---|---|
| version increase simulation | Second Tuesday after version increase | The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let probe-scraper alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.
Simulate the code in mozilla-central with the next higher version number. |
| Create new beta simulation document | Monday on which version got increased | Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management. |
| Request new version to be added to Bugzilla | Monday before next version increase | The Gecko and Firefox version will get increased and bugs cannot be set as fixed in the new version with the Bugherder tool until the version got added to bugzilla.mozilla.org. The new version gets requested one week in advance to allow easier coordination with the Bugzilla team. |
| Add new version to tree statistics | Monday on which version got increased | Tool which shows how often and how long the trees got closed |
| Last cycle quality metrics | Monday on which version got increased | Accumulate weekly quality metrics for last release cycle |
| Increase simulation version numbers | Monday on which version got increased | Increment version numbers for central-as-beta and beta-as-release simulations at Sheriffing/How To/Beta simulations |
| New Nightly respin statistics | Monday on which version got increased | Set up new sheet to track backouts from mozilla-central and Nightly respins |
| Land patches | daily | Check for patches waiting to land |
