Sheriffing/Schedule for Tasks performed by Code Sheriffs: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(added request of new version to be added to Bugzilla and handling of expiring telemetry probes)
(update tasks Aryx)
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
Version: 3 (Last Updated 2019-09-25)
Version: 8 (Last Updated 2021-08-10)
{| class="wikitable"
{| class="wikitable"
|-
|-
Line 18: Line 18:
||Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
||Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
|- style="vertical-align:top"
|- style="vertical-align:top"
| Land patches and do uplifts to beta || Every 2h
| Merge autoland tree to mozilla-central || Mo-Fr every 6h
||Check for patches waiting to land (have keyword “checkin-needed” at Bugzilla) or to be uplifted from central to beta. Beta patches shall have landed and build and test coverage before 9am Pacific Time on Mondays and Thursdays when betas are built. Release Management might request to land more on short notice.
|- style="vertical-align:top"
| Merge integration trees (no central to autoland)|| Mo-Fr every 6h
Sa+Su: every 12h
Sa+Su: every 12h
|| Merge autoland to mozilla-central and mozilla-central to autoland if necessary
|| Merge autoland to mozilla-central and mozilla-central to autoland if necessary
|- style="vertical-align:top"
|- style="vertical-align:top"
| central-as-beta simulations || Daily
| central-as-beta simulations || Early Beta: every day
Late Beta: Tue+Fri
||Simulate the code in mozilla-central as mozilla-beta
||Simulate the code in mozilla-central as mozilla-beta
|- style="vertical-align:top"
|- style="vertical-align:top"
| beta-as-release simulation || Weekly during the Thursday night shift
| beta-as-release simulation || Thursdays
||Simulate the code in mozilla-beta as mozilla-release
||Simulate the code in mozilla-beta as mozilla-release, skip in last week of development cycle when beta already got merged to release
|- style="vertical-align:top"
| Intermittent triage || Weekend
||Frequent failure escalation or reminders to developers
|- style="vertical-align:top"
|- style="vertical-align:top"
| Clean up list of bugs which don’t get suggested || Weekly
| Clean up list of bugs which don’t get suggested || Weekly
Line 36: Line 37:
# If the bug has been resolved as incomplete, remove it from the document.
# If the bug has been resolved as incomplete, remove it from the document.
# If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.
# If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.
|}
====Release graphs====
In the nights from
* Sunday to Monday
* Tuesday to Wednesday
* Thursday to Friday
the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).
These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).
Exceptions are:
* Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
* week before release: release candidates get built instead and release management starts these manually.
Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.
* Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty] channel on Matrix if they continue to fail.
* 5h after the release graphs got scheduled and posted in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty channel] (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
** The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
** Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the [https://chat.mozilla.org/#/room/#releaseduty:mozilla.org releaseduty channel] and create a bug if there is no reply in 15 minutes.
*** These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
*** Needinfo the triage owner and the [[Release_Management/Release_owners|release manager who owns the release]].
** Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
** The same process also applies to "rapid" releases and "ESR" releases.
====Sheriffing and quality/stability related tasks executed by one person (Aryx)====
{| class="wikitable"
|-
! Task !! Schedule !! Details
|- style="vertical-align:top"
|- style="vertical-align:top"
| Request new version to be added to Bugzilla || Monday before next version increase || The Gecko and Firefox version will get increased and bugs cannot be set as fixed in the new version with the [https://bugherder.mozilla.org/ Bugherder] tool until the version got added to bugzilla.mozilla.org. The new version gets requested one week in advance to allow easier coordination with the Bugzilla team.
| version increase simulation || Second Tuesday after version increase || The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let [https://github.com/mozilla/probe-scraper probe-scraper] alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.
[https://bugzilla.mozilla.org/show_bug.cgi?id=1606669 Example bug]
Simulate the code in mozilla-central with the next higher version number.
# Clone [https://bugzilla.mozilla.org/buglist.cgi?list_id=15051791&short_desc=add%20gecko&classification=Client%20Software&classification=Developer%20Infrastructure&classification=Components&classification=Server%20Software&classification=Other&chfield=%5BBug%20creation%5D&query_format=advanced&chfieldfrom=80d&short_desc_type=allwordssubstr&component=Administration&product=bugzilla.mozilla.org the bug which added the latest version to Bugzilla].
|- style="vertical-align:top"
| Create new beta simulation document|| Monday on which version got increased
||Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management.
|- style="vertical-align:top"
| Add new version to tree statistics || Monday on which version got increased
||Tool which shows how often and how long the trees got closed
* [https://github.com/Archaeopteryx/treestats code]
* [https://archaeopteryx.github.io/treestats/ tool]
|- style="vertical-align:top"
|- style="vertical-align:top"
| version increase simulation || First Tuesday after version increase || The first time beta simulations are done after the version number got increased (usually a Tuesday), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.
| Increase simulation version numbers || Monday on which version got increased
Simulate the code in mozilla-central with the next higher version number.
||Increment version numbers for central-as-beta and beta-as-release simulations at [[Sheriffing/How To/Beta simulations]]
|- style="vertical-align:top"
|- style="vertical-align:top"
| create bugs for expiring telemetry probes || First Wednesday after version increase
| Land patches || daily
||Not every expiring browser measurement will cause test breakage. Create a bug for each group of expiring telemetry probes (group by the bugs which added or extended them according to the history annotation on the left in searchfox). Find them with an adjusted version of [https://searchfox.org/mozilla-central/search?q=expir.*74&regexp=true&path=toolkit%2Fcomponents%2Ftelemetry%2F this query].
||Check for patches waiting to land
|}
|}

Latest revision as of 14:02, 30 September 2024

Version: 8 (Last Updated 2021-08-10)

Task Schedule Details
Watch trees Always Watch the following trees for failures, classify them, do backouts and escalate infra issues:
  • trunk (mozilla-central, autoland)
  • mozilla-beta
  • mozilla-release
  • mozilla-esr trees (all of them which are available in Treeherder; often there is only 1, but for some time 2 when their cycles overlap)

Open them up to the oldest push with unclassified failures (excluding backfills and retriggers added >4h after the push) or running jobs.

Email reading Start of shift, every 30 minutes Help other people by answering needinfo requests etc.
Read new, not suggested bugs Start of shift Get used to new frequent failures created since the last shift and which Treeherder doesn’t suggest. These are added at the top and further down.
Merge autoland tree to mozilla-central Mo-Fr every 6h

Sa+Su: every 12h

Merge autoland to mozilla-central and mozilla-central to autoland if necessary
central-as-beta simulations Early Beta: every day

Late Beta: Tue+Fri

Simulate the code in mozilla-central as mozilla-beta
beta-as-release simulation Thursdays Simulate the code in mozilla-beta as mozilla-release, skip in last week of development cycle when beta already got merged to release
Intermittent triage Weekend Frequent failure escalation or reminders to developers
Clean up list of bugs which don’t get suggested Weekly Check if bugs are still active or needed for beta. Preferable on the weekend when there is more time for this.
  1. Open the Bugzilla links from the document.
  2. If the bug has been resolved as incomplete, remove it from the document.
  3. If it is resolved fixed but the widget showing the number of classifications per day still shows activity, it is used for non-trunk trees (e.g. beta, release, esrs). Keep it in that case.

Release graphs

In the nights from

  • Sunday to Monday
  • Tuesday to Wednesday
  • Thursday to Friday

the tasks which are required to ship beta builds to users run (e.g. creating builds other than US English, creating updates and verifying their correctness).

These tasks start at 21:00 UTC (= 23:00 Romanian time in winter, 00:00 in summer).

Exceptions are:

  • Sunday to Monday when Monday is merge day: the beta build starts after the merge on Monday.
  • week before release: release candidates get built instead and release management starts these manually.

Sometimes these beta builds are not part of the last 10 pushes which are shown by default by Treeherder. People starting a shift during which such a "release graph" for beta is supposed to run shall load pushes for mozilla-beta until they see a push with changes affecting also the US English Firefox (= usually the pushes from release management). These should get new running tasks after the aforementioned times. For the Sunday to Monday shift, this can be a push from Friday.

  • Failing tasks ("UV", "c-up", "L10n" etc.) should be rerun and escalated in the releaseduty channel on Matrix if they continue to fail.
  • 5h after the release graphs got scheduled and posted in the releaseduty channel (4am Romanian time in winter, 5am in summer), check the link from the channel if all tasks completed successfully (if the requirements are fulfilled for the build to ship).
    • The page can slow down the browser: If you notice this, just open the page, wait for it to load the data about all the tasks belonging to the release graph, check if there are tasks which are still not complete yet, and close the page.
    • Not every task is shown by Treeherder but this Taskcluster view is aware of all. If there are still unexpected incomplete tasks 5h after the start of the release graph, mention it in the releaseduty channel and create a bug if there is no reply in 15 minutes.
      • These usually should be filed against one of the Release Engineering :: Release Automation components in Bugzilla.
      • Needinfo the triage owner and the release manager who owns the release.
    • Be aware 2 release graphs have to be checked per beta release (the "Beta" branded one and the one for the "Developer Edition").
    • The same process also applies to "rapid" releases and "ESR" releases.

Sheriffing and quality/stability related tasks executed by one person (Aryx)

Task Schedule Details
version increase simulation Second Tuesday after version increase The first time a version increase simulations is done is the second Tuesday after the version number got increased (to let probe-scraper alert for expiring probes before and reduced the failures in the version increase simulation), later as needed - at least a week before the next version increase to verify fixes worked and no new issues have been added.

Simulate the code in mozilla-central with the next higher version number.

Create new beta simulation document Monday on which version got increased Clone the current beta simulation document, adjust versions and dates and send it to code sheriffs and release management.
Add new version to tree statistics Monday on which version got increased Tool which shows how often and how long the trees got closed
Increase simulation version numbers Monday on which version got increased Increment version numbers for central-as-beta and beta-as-release simulations at Sheriffing/How To/Beta simulations
Land patches daily Check for patches waiting to land
Retrieved from "https://wiki.mozilla.org/index.php?title=Sheriffing/Schedule_for_Tasks_performed_by_Code_Sheriffs&oldid=1252000"