Sheriffing/How To/Backouts

When a patch is landed that causes build or test failures, it must be backed out to restore the tree to a passing or green state again.

How to perform backouts

There are a few things to check out which can indicate the necessity of a backout:

1. there are ES (linting opt) failures: close the tree, backout, inform the developer
2. Requested by the developer
3. The push has many bustages
4. The push has a high failure rate
5. The failure occurs on multiple pushes
6. The failure is not assigned to a bug

Before the backout, we should close the tree from which we’ll backout in order to avoid race pushing.

With unified repos, backouts are now easier to perform. Please follow the instructions for backing out from unified repos.

Steps:

1. close the tree
2. cd mozilla-unified //access the mozilla-unified directory
3. hg pull <repo> //download repo
4. hg update <repo> //update repo
5. identify the revision we want to backout
6. hg oops -er <revision> //run the backout command, for a single changeset. If the backout impacts multiple changesets, use hg oops -esr <revision>::<revision> instead.If there are two changesets with other unrelated changesets inbetween, usehg oops -esr <revision>+<revision>
7. open the text editor to add the description of the backout “Backed out changeset <revision> (bug X) for <insert reason here>”. If the repo(s) are closed, then we also need to add “on a CLOSED TREE”. If the repo(s) are approval-only we need to add a=backout. The reason should mention the build or test suite (and in the latter case also the failing test) and the affected platforms if it doesn't fail on every.
8. hg push -r . <repo> // push the changes

Example on how to backout a single revision

Note: The example below is for autoland, but the same steps should be followed for any repos

1. close the tree
2. cd mozilla-unified
3. hg pull autoland
4. hg update autoland
5. identify the revision we want to backout (e.g. def0af88e262)
6. hg oops -er def0af88e262
7. open the text editor to add the description of the backout, e.g.: “Backed out changeset def0af88e262 (bug 1359017) for <insert reason here> CLOSED TREE”.
8. hg push -r . autoland

Example on how to backout multiple revisions

Note: The example below is for autoland, but the same steps should be followed for any repos

1. close the tree
2. cd mozilla-unified
3. hg pull autoland
4. hg update autoland
5. identify the push we want to backout. E.g.: 478cffed4b5f, daea2bcda8cb, 9abaf5195566, 2f665a0a379f
6. hg oops -esr 2f665a0a379f::478cffed4b5f
7. open the text editor to add the description of the backout, e.g.: “Backed out X changesets (bug 1359017) for <insert reason here> CLOSED TREE”.
8. hg push -r . autoland

Best Practices and Communication

Based on Feedback from the Sheriff Survey, when you have to do a backout, here are some best practices:

• Ping the Developer in IRC if possible and when the developer is around, this gives the developer a chance to fix the problem and so avoid a backout. Some of the sheriffs use a 5 minute rule for the waiting of a response of the developer. Note: this is done as a courtesy to developers. You can and should backout patches if you feel it's being abused, e.g. the developer is not using the Try server
• After you backed a change out, update the bug in Bugzilla:
  • Mention the reason for the backout.
  • Link to the backout commit starting with https://hg.mozilla.org
  • Link to the push which got backed out, use a link which also shows classified failures. That way developers can check what platforms are affected and if there are more issues.
  • Link to failure log of a failed log for easy access.
  • Snippet of the failure message: This might provide insight if the developer can fix it quickly.
  • Set the NEEDINFO flag in bugzilla. This is very important! Developers get a lot of bugmail and backout comments can be missed. Please make sure that you set a NEEDINFO to the developer in the bug when doing a backout. This is usually done by selecting "assignee" from the needinfo dropdown near the bottom.
• Make sure that the failures that lead to the backout are starred in Treeherder. This helps the next sheriff on duty.
• First a change might be backed out for one issue but later a longer running task also fails due to those changes but maybe with a completely unrelated test. Update the backed out bug about this. Use a new comment, don't edit the previous backout comment because there is no bug mail for that.

Backouts from mozilla-central

If a developer asks for a backout of patch which has already been merged to central, code sheriffs should ask them if the fix requires to ship a new Nightly as soon as possible. If yes, back it out from central, else request new Nightlies and merge back to autoland, else just back it out from autoland. This guides the developer better than the question if the backout should be done on autoland or central.

How to test a backout on the Try server

If there is uncertainity if a backout will fix an issue, this can be tested on the Try server where changes are tested but never committed to production repositories.

1. First backout the changeset(s)
   hg oops -er or hg oops -esr
2. At https://mozilla-releng.net/trychooser/ choose the build types, the platforms, test suites etc. that you need to test.
   Example:
   
3. For the backout’s commit message, we will use what is written in the Computed Syntax command field. Example:
   
4. To push to try, use the following command:
   hg push -f ssh://hg.mozilla.org/try
5. After the backout was pushed to Try, we need to drop the commit from history, for every single changeset that was tested:
   Run hg histedit and replace pick with d.

Rejected backout pushes

By default, the server will reject changes to some folder or file types and a keyword needs to be part of the commit message to let it pass. This has also been used when the change landed (can also be in a different line of the commit message than the first one - in that case Treeherder won't show it) and is usually written in UPPERCASE.

Examples:

• changesets contain changes to protected nsprpub/ directory: Commit message requires UPGRADE_NSPR_RELEASE
• changesets contain changes to protected security/nss/ directory: Commit message requires UPGRADE_NSS_RELEASE

Useful commands

If you want to find all changesets which landed for a bug number, e.g. in multiple commits:

hg log -k "bug number"

E.g.

hg log -k "1234567"

Backouts of bugless commits

If the backed out commit has no bug number:

1. Create a bug in which the failure and the backout is mentioned. Bugless commits should not be able to cause failures or bustage.
2. Needinfo the person who pushed the failing change. If it was pushed by a bot, needinfo the triage owner of the modified files.