Sheriffing/Sheriff Duty

What's a sheriff, anyways?

Every day a person from the mozilla community is selected to watch over the build tree. Make sure red gets traction, call people on the phone, etc. Send mail to the sheriffs alias to reach the set of eligible sheriffs.

How should I get ahold of people?

Try irc.mozilla.org, #developers and #build.

You may also want to contact the RelEng Sheriff

How do I change the mozilla tinderbox message of the day?

• You can set the message using treestatus, see: https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree

What kind of comments might the sheriff add to Tinderbox?

The sheriff has broad discretion to add tinderbox comments related to the Mozilla builds, tree opening, blockers etc.

Developers wishing changes to the Mozilla tinderbox messages should communicate with the sheriff so that the changes can be coordinated.

When should I close the mozilla tree?

The tree should be closed due to:

• system outages that prevent the tinderbox machines from running the necessary tests (compilation, function, performance, memory) on changes that are checked in
• bustage (compilation errors, test failures, or significant performance/memory test regressions) that prevents tinderbox machines from determining if new changes cause further regressions
• bugs that are bad enough to prevent the builds from being used as dogfood

Note that a closed tree is different from a restricted tree, where checkins require additional approval or additional criteria due to an impending release.

If you close the tree, please record the time and the reasons why on the Tree Closures page.

When should I reopen the tree after closing it?

When the tree has been closed due to build bustage, performance regressions, or test regressions, it should be reopened once:

• the sheriff has confidence that those regressions are diagnosed and being fixed (preferably fixed already), and
• tinderbox is clear enough (or expected to be in the cycle that current checkins would go into) that any new regressions that should be caught will be.

This does not mean that all boxes need to cycle green again before reopening. However, if there were a large number of checkins since things started going bad, it is often good to wait for test boxes to cycle green again to make sure there aren't regressions caused by what has already landed.

If there has been a lot of tree closure recently, continuing to hold the tree closed increases the chances that there will be a mess when it reopens and everybody lands at once. Tree closures should therefore be kept to the minimum necessary, especially when there have been many recent closures or there is an upcoming deadline.

Please make sure to update the Tree Closures page with the time of the opening, as well as any notes that would be useful for a postmortem about that particular closure.

How do I open/close the mozilla tree?

• If the tree is being closed for anything other than red or short-lived orange, file a bug about the problem for which it is being closed:
  • If it's a performance or unit test regression, the bug should say which tinderboxes, when, and give an URL to the list checkins that may have caused the regression.
  • If it's a functional regression, the bug should have steps to reproduce, the dates of the nightly builds immediately surrounding the regression (or narrower window if known), and an URL to the list of checkins that could have caused the regression.
• See https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree
• Update the tree closure log

What do I do about regressions?

See the official regression policy document: Regression Policy

Who are the sheriffs?

See https://wiki.mozilla.org/Sheriffing

How to monitor the tree status?

Use Treeherder.

Is there an easy way to monitor all performance graphs?

Yes! Thanks to Johnathan Nightingale, we now have a performance dashboard.

See also the performance monitoring wiki.

How do I cancel/restart a job?

You can use self-serve as a means of cancelling/restarting a build/test job. You can also restart/cancel jobs from Treeherder by clicking on the job in question, and pressing either the retrigger or cancel buttons in the job details panel.

How do I back out changes from Mercurial?

See Mercurial FAQ: Backing out changes.

How do I back out changes from Gaia?

Gaia changes may need to be backed out if a Gaia Pushbot change caused tests to go orange. When that happens, you should follow these steps to back out the relevant commit(s):

• Examine the changes to gaia.json, which may refer to one or more github commits. For example, https://hg.mozilla.org/projects/birch/rev/377785a4134c shows two gaia commits: https://hg.mozilla.org/integration/gaia-central/rev/c61d272d29ae and https://hg.mozilla.org/integration/gaia-central/rev/8515ff8e847a.
• Find the github commit which matches the first hg commit. In this case, https://hg.mozilla.org/projects/birch/rev/377785a4134c maps to https://github.com/mozilla-b2g/gaia/commit/f657ccca8e25486f7ca44e5472d9f20ff4a7b4f4.
• If there is only one gaia commit in gaia.json, it can simply be reverted and then pushed:

 git revert f657ccca8e25486f7ca44e5472d9f20ff4a7b4f4
 git push

• If there are multiple commits in gaia.json, then you should determine which commits were done against the gaia repo. You can do this with:

 git log f657ccca8e25486f7ca44e5472d9f20ff4a7b4f4 --first-parent

For each commit which is part of this git log and is also part of the gaia.json change you are reverting, do a git revert:

 git revert -m 1 f657ccca8e25486f7ca44e5472d9f20ff4a7b4f4
 git revert -m 1 <next commit>
 git push

There is no need to revert gaia.json directly, as changes to gaia will automatically update this file.

How do I clobber a build machine or entire branch?

See Clobbering the Tree

What about mozilla-inbound?

mozilla-inbound has its own set of sheriffs for now, read about Inbound Sheriff Duty. Basically, that tree is always open for landing, and its sheriffs deal with bustages and regressions by backing out the offending patches.