Sheriffing/Schedule for Tasks performed by Code Sheriffs

From MozillaWiki
Jump to: navigation, search

Version: 7 (Last Updated 2021-01-26)

Task Schedule Details
Watch trees Always Watch the following trees for failures, classify them, do backouts and escalate infra issues:
  • trunk (mozilla-central, autoland)
  • mozilla-beta
  • mozilla-release
  • mozilla-esr trees (all of them which are available in Treeherder; often there is only 1, but for some time 2 when their cycles overlap)

Open them up to the oldest push with unclassified failures (excluding backfills and retriggers added >4h after the push) or running jobs.

Email reading Start of shift, every 30 minutes Help other people by answering needinfo requests etc.
Read new, not suggested bugs Start of shift Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
Merge autoland tree to mozilla-central Mo-Fr every 6h

Sa+Su: every 12h

Merge autoland to mozilla-central and mozilla-central to autoland if necessary
central-as-beta simulations Weekends Simulate the code in mozilla-central as mozilla-beta
Clean up list of bugs which don’t get suggested Weekly Check if bugs are still active or needed for beta. Preferable on the weekend when there is more time for this.
  1. Open the Bugzilla links from the document.
  2. If the bug has been resolved as incomplete, remove it from the document.
  3. If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.

Release graphs

In the nights from

  • Sunday to Monday
  • Tuesday to Wednesday
  • Thursday to Friday

the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).

These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).

Exceptions are:

  • Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
  • week before release: release candidates get built instead and release management starts these manually.

Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.

  • Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the releaseduty channel on Matrix if they continue to fail.
  • 5h after the release graphs got scheduled and posted in the releaseduty channel (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
    • The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
    • Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the releaseduty channel and create a bug if there is no reply in 15 minutes.
      • These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
      • Needinfo the triage owner and the release manager who owns the release.
    • Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
    • The same process also applies to "rapid" releases and "ESR" releases.

Sheriffing and quality/stability related tasks executed by one person (Aryx)

Task Schedule Details
central-as-beta simulations Mon-Fri Simulate the code in mozilla-central as mozilla-beta
beta-as-release simulation Thursdays Simulate the code in mozilla-beta as mozilla-release
version increase simulation Second Tuesday after version increase The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let probe-scraper alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.

Simulate the code in mozilla-central with the next higher version number.

beta uplifts on demand from Release Management Check for patches to be uplifted from central to beta. Beta patches shall have landed and build and test coverage before 2am UTC on Mondays, Wednesdays and Fridays when betas are built. Release Management might request to land more on short notice.
Create new beta simulation document Monday on which version got increased Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management.
Request new version to be added to Bugzilla Monday before next version increase The Gecko and Firefox version will get increased and bugs cannot be set as fixed in the new version with the Bugherder tool until the version got added to bugzilla.mozilla.org. The new version gets requested one week in advance to allow easier coordination with the Bugzilla team.

Example bug

  1. Clone the bug which added the latest version to Bugzilla.
Add new version to tree statistics Monday on which version got increased Tool which shows how often and how long the trees got closed
Last cycle quality metrics Monday on which version got increased Accumulate weekly quality metrics for last release cycle
Increase simulation version numbers Monday on which version got increased Increment version numbers for central-as-beta and beta-as-release simulations at Sheriffing/How To/Beta simulations
Last cycle central-as-beta metrics Monday on which version got increased or soon after Accumulate central-as-beta and version increase metrics: how many issues got detected, what is the average time to fix them broken down by build vs test failure
New Nightly respin statistics Monday on which version got increased Set up new sheet to track backouts from mozilla-central and Nightly respins
Land patches several times per day Check for patches waiting to land