|
|
| (5 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| = Autoland =
| | The current incarnation of automatic code landings at Mozilla is [https://lando.services.mozilla.com Lando]. |
| | |
| == Introduction ==
| |
| Autoland is the name of a project which will enable automatic landing of changesets to mozilla trees, for instance from a MozReview repository to try or to mozilla-inbound after a successful try run. This will allow developers to land changesets "automatically", with many fewer manual steps than are now required.
| |
| | |
| == History ==
| |
| Several past projects have used the name "Autoland" and have had roughly the same goal. These have mostly failed to achieve deployment due to difficulty in being able to meaningfully determine whether a patch or changeset has "passed" a try run, due to the number of intermittent oranges which are likely to affect a given try push.
| |
| | |
| These projects include:
| |
| | |
| * [[BMO/AutoLand]]
| |
| * [[ReleaseEngineering:Autoland]] and related [https://github.com/rail/build-autoland/ github repo]
| |
| | |
| In Q4 of 2014, a prototype system was built with the goal of automatically landing changes from try to mozilla-inbound after a successful try run. It used pulse to monitor try runs pushed with a special syntax. Failed jobs were retriggered to determine if the failure was due to an intermittent or not. If successful, the transplant service under development by RelEng was used to transplant the results from try to another repository.
| |
| | |
| == Non-goals ==
| |
| * This version of Autoland works on changesets, and will not retrieve patches out of Bugzilla.
| |
| * This version of Autoland will be designed to work against trunk branches, and will not be designed to assist in uplifting changesets. Future versions of Autoland may add that capability.
| |
| | |
| == Workflow ==
| |
| | |
| | |
| | |
| == Components ==
| |
| | |
| === Autoland Service [owner: dminor] ===
| |
| This is the primary component of Autoland. This service monitors pulse for autoland jobs submitted to try, monitors such try jobs for completion using the buildapi, queries the buildapi for job status, and retriggers failing jobs. The project repo is located at [1].
| |
| | |
| It also submits comments to Bugzilla when the Autoland status of a try push changes, and lands patches on inbound trees if all success criteria are met. Patches will be landed using the transplant tool [2].
| |
| | |
| [1] https://github.com/dminor/autoland
| |
| | |
| [2] https://github.com/laggyluke/transplant
| |
| | |
| === Treeherder [owner: dminor] ===
| |
| Treeherder will be utilized in the following ways:
| |
| | |
| # Treeherder will expose a special Autoland view of try jobs which need sheriff attention; this view will include a button (available only to privileged users) that will allow a pending Autoland push to be landed or rejected.
| |
| | |
| See {{bug|1058864}}
| |
| | |
| === Releng Buildapi ===
| |
| This api (which exists already) will be used to determine when Autoland try jobs are finished, and will be used to retrigger failing jobs.
| |
| | |
| docs: [[ReleaseEngineering/Applications/BuildAPI]]
| |
| | |
| | |
| == Roadmap ==
| |
| | |
| === Milestone 2 ===
| |
| | |
| Integration with MozReview to enable landing of patches from MozReview to Try from within MozReview with monitoring and reporting of results
| |
| | |
| target delivery date: 2015Q1
| |
| | |
| <bugzilla>
| |
| {
| |
| "whiteboard": "[autoland m2]",
| |
| "component": "Autoland"
| |
| }
| |
| </bugzilla>
| |
| | |
| === Milestone 1 ===
| |
| | |
| A prototype which can be tested end-to-end by pushing to try with a specific syntax.
| |
| | |
| target delivery date: 2014Q4
| |
| | |
| status: We deployed an initial version of Autoland that is able to listen to try pushes and utilize the RelEng Transplant tool to transplant successful try pushes from Try to a clone of mozilla-central hosted on a staging instance of the Transplant tool. Transplant is still under active development and is not ready to be deployed, which necessitated testing against a staging instance and a lcocal clone of mozilla-central.
| |
| | |
| When pushing autoland patches to try it is necessary that the head patch be empty and that the commit message contains the try syntax along with the string '--autoland'.
| |
| | |
| An example of a summary for an autoland push is:
| |
| | |
| try: -b do -p all -u all -t none --autoland
| |
| Bug XXXXXX - Do interesting stuff; r=yyy
| |
| | |
| The autoland tool does not mandate a minimal set of tests for a change or check that the patch has been properly reviewed. In order to keep autoland as simple as possible it is assumed that these policies will be enforced elsewhere, for instance, in Mozreview. A check is done to verify that the person who pushed to try is a member of the appropriate LDAP group to land on the destination repository, i.e. scm level 3 in order to land on mozilla-inbound.
| |
| | |
| A single failure is allowed for each build and test. In this case, autoland automatically retriggers the job to see if it will succeed on the second attempt. If it succeeds on the second run, it is assumed that the failure was intermittent and that the patch can be landed. If two failures occur for any build or test, the autoland request fails.
| |
| | |
| If the autoland request succeeds, the transplant tool is used to land the changes. Whether or not the request succeeds, the bugzilla bug corresponding to the request is updated.
| |
| | |
| The proposed workflow for Autoland is as follows:
| |
| | |
| 1 - ReviewBoard or developer submits patch/changeset to try with a specific syntax, maybe --autoland.<br/>
| |
| 1a - If the try run is being requested by ReviewBoard, check that the requester (not the patch author) has commit access level 3, and if not, reject with an appropriate error message. If the try run has been requested directly, this check isn't performed until step 3.
| |
| | |
| 2 - A try run is performed (with a subset of tests if specified by the submitter).<br/>
| |
| 2a - When performed via ReviewBoard, we should encourage targeted try runs in order to conserve resources.<br />
| |
| 2b - When performed via ReviewBoard, we should force the try run against the tip of mozilla-inbound, in order to prevent the case where a changeset may pass a try run against an older gecko commit that the review was created against, but fail when landing on inbound.
| |
| | |
| 3 - An Autoland service somewhere is consuming the pulse stream for the try repo, notices an Autoland job has been submitted to try and starts monitoring it.<br />
| |
| 3a - If the try run was submitted directly (not via ReviewBoard), the Autoland service should check the commit level access of the submitter, and reject the Autoland attempt if the submitter does not have commit access level 3.
| |
| | |
| 4 - When no pending jobs remain, the Autoland service looks for failed jobs; if it finds any it retriggers them.<br />
| |
| 4a - Autoland will determine when all jobs are complete using the buildapi at [https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-central/rev/2d88803a0b9c/is_done]<br />
| |
| 4b - Autoland will determine job status by querying the buildapi<br />
| |
| 4c - Autoland will retrigger failed jobs using the buildapi<br />
| |
| 4d - Initially, Autoland will retrigger failed jobs only once before taking action; this can be tweaked in the future as needed if we want to make it smarter in hunting for frequent intermittent failures introduced by the changeset.
| |
| | |
| 5 - When Autoland service gets results for the retriggered jobs:<br/>
| |
| 5a - If all retriggered jobs pass, it lands the patch in-tree (m-i by default, by developer can request another tree using some TBD try syntax)<br/>
| |
| 5b - If patch no longer cleanly applies, Autoland rejects with a message on the bug<br/>
| |
| 5c - If the retriggered jobs fail, Autoland will annotate the push in Treeherder
| |
| | |
| 6 - Sheriffs periodically monitor the Autoland queue in Treeherder for jobs that need human inspection, and flag them as ok to land or not ok to land, possibly after additional manual retriggers.<br/>
| |
| 6a - If ok to land, autoland lands the patch in-tree. We may coalesce pending autoland commits and land them in batches periodically, like once per hour, ala Bumper Bot.<br/>
| |
| 6b - If not ok to land, Autoland notifies the developer of the problem via bugzilla.
| |
| | |
| | |
| == Bugs ==
| |
| Bugs should be filed in Bugzilla under Tree Management:Autoland.
| |
| | |
| <bugzilla>
| |
| {
| |
| "quicksearch": "status:new,assigned,reopened,unconfirmed",
| |
| "product": "Tree Management",
| |
| "component": "Autoland"
| |
| }
| |
| </bugzilla>
| |