Autoland

Introduction

Autoland is the name of a project which will enable automatic landing of changesets to mozilla trees, for instance from a MozReview repository to try or to mozilla-inbound after a successful try run. This will allow developers to land changesets "automatically", with many fewer manual steps than are now required.

History

Several past projects have used the name "Autoland" and have had roughly the same goal. These have mostly failed to achieve deployment due to difficulty in being able to meaningfully determine whether a patch or changeset has "passed" a try run, due to the number of intermittent oranges which are likely to affect a given try push.

These projects include:

BMO/AutoLand
ReleaseEngineering:Autoland and related github repo

In Q4 of 2014, a prototype system was built with the goal of automatically landing changes from try to mozilla-inbound after a successful try run. It used pulse to monitor try runs pushed with a special syntax. Failed jobs were retriggered to determine if the failure was due to an intermittent or not. If successful, the transplant service under development by RelEng was used to transplant the results from try to another repository.

Non-goals

This version of Autoland works on changesets, and will not retrieve patches out of Bugzilla.
This version of Autoland will be designed to work against trunk branches, and will not be designed to assist in uplifting changesets. Future versions of Autoland may add that capability.

Workflow

Components

Autoland [owner: dminor]

This is the primary component of Autoland. It lands revisions from one repository to another and monitors associated test runs, retriggered failing jobs as necessary. The project repo is part of the v-c-t repo at [1] in the autoland directory.

[1] https://hg.mozilla.org/hgcustom/version-control-tools/autoland

Autoland Pulse Listener

This component listens for build start and build complete messages over Pulse [1] and uses this information to update status about test runs that are being monitored by Autoland.

[1] http://pulse.mozilla.org

Autoland REST Interface

This component provides a web interface to allow requests for revisions to be landed from one repository to another.

MozReview [owner: mdoglio]

MozReview will be used to provide a UI for landing changes from MozReview repositories to Try and to retrieve and display data from Treeherder for the resulting test run.

See bug 1109218

Roadmap

Milestone 2

Integration with MozReview to enable landing of patches from MozReview to Try from within MozReview with monitoring and reporting of results

target delivery date: 2015Q1

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Milestone 1

A prototype which can be tested end-to-end by pushing to try with a specific syntax.

target delivery date: 2014Q4

status: We deployed an initial version of Autoland that is able to listen to try pushes and utilize the RelEng Transplant tool to transplant successful try pushes from Try to a clone of mozilla-central hosted on a staging instance of the Transplant tool. Transplant is still under active development and is not ready to be deployed, which necessitated testing against a staging instance and a lcocal clone of mozilla-central.

When pushing autoland patches to try it is necessary that the head patch be empty and that the commit message contains the try syntax along with the string '--autoland'.

An example of a summary for an autoland push is:

   try: -b do -p all -u all -t none --autoland
   Bug XXXXXX - Do interesting stuff; r=yyy

The autoland tool does not mandate a minimal set of tests for a change or check that the patch has been properly reviewed. In order to keep autoland as simple as possible it is assumed that these policies will be enforced elsewhere, for instance, in Mozreview. A check is done to verify that the person who pushed to try is a member of the appropriate LDAP group to land on the destination repository, i.e. scm level 3 in order to land on mozilla-inbound.

A single failure is allowed for each build and test. In this case, autoland automatically retriggers the job to see if it will succeed on the second attempt. If it succeeds on the second run, it is assumed that the failure was intermittent and that the patch can be landed. If two failures occur for any build or test, the autoland request fails.

If the autoland request succeeds, the transplant tool is used to land the changes. Whether or not the request succeeds, the bugzilla bug corresponding to the request is updated.

The proposed workflow for Autoland is as follows:

1 - ReviewBoard or developer submits patch/changeset to try with a specific syntax, maybe --autoland.
1a - If the try run is being requested by ReviewBoard, check that the requester (not the patch author) has commit access level 3, and if not, reject with an appropriate error message. If the try run has been requested directly, this check isn't performed until step 3.

2 - A try run is performed (with a subset of tests if specified by the submitter).
2a - When performed via ReviewBoard, we should encourage targeted try runs in order to conserve resources.
2b - When performed via ReviewBoard, we should force the try run against the tip of mozilla-inbound, in order to prevent the case where a changeset may pass a try run against an older gecko commit that the review was created against, but fail when landing on inbound.

3 - An Autoland service somewhere is consuming the pulse stream for the try repo, notices an Autoland job has been submitted to try and starts monitoring it.
3a - If the try run was submitted directly (not via ReviewBoard), the Autoland service should check the commit level access of the submitter, and reject the Autoland attempt if the submitter does not have commit access level 3.

4 - When no pending jobs remain, the Autoland service looks for failed jobs; if it finds any it retriggers them.
4a - Autoland will determine when all jobs are complete using the buildapi at [1]
4b - Autoland will determine job status by querying the buildapi
4c - Autoland will retrigger failed jobs using the buildapi
4d - Initially, Autoland will retrigger failed jobs only once before taking action; this can be tweaked in the future as needed if we want to make it smarter in hunting for frequent intermittent failures introduced by the changeset.

5 - When Autoland service gets results for the retriggered jobs:
5a - If all retriggered jobs pass, it lands the patch in-tree (m-i by default, by developer can request another tree using some TBD try syntax)
5b - If patch no longer cleanly applies, Autoland rejects with a message on the bug
5c - If the retriggered jobs fail, Autoland will annotate the push in Treeherder

6 - Sheriffs periodically monitor the Autoland queue in Treeherder for jobs that need human inspection, and flag them as ok to land or not ok to land, possibly after additional manual retriggers.
6a - If ok to land, autoland lands the patch in-tree. We may coalesce pending autoland commits and land them in batches periodically, like once per hour, ala Bumper Bot.
6b - If not ok to land, Autoland notifies the developer of the problem via bugzilla.

Bugs

Bugs should be filed in Bugzilla under Tree Management:Autoland.