|
|
| (8 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| = Autoland =
| | The current incarnation of automatic code landings at Mozilla is [https://lando.services.mozilla.com Lando]. |
| | |
| == Introduction ==
| |
| Autoland is the name of a project which will enable automatic landing of changesets after a successful try run. This will allow developers to land changesets "automatically", with many fewer manual steps than are now required.
| |
| | |
| == History ==
| |
| Several past projects have used the name "Autoland" and have had roughly the same goal. These have mostly failed to achieve deployment due to difficulty in being able to meaningfully determine whether a patch or changeset has "passed" a try run, due to the number of intermittent oranges which are likely to affect a given try push.
| |
| | |
| These projects include:
| |
| | |
| * [[BMO/AutoLand]]
| |
| * [[ReleaseEngineering:Autoland]] and related [https://github.com/rail/build-autoland/ github repo]
| |
| | |
| == Non-goals ==
| |
| * This version of Autoland works on changesets, and will not retrieve patches out of Bugzilla.
| |
| * This version of Autoland will be designed to work against trunk branches, and will not be designed to assist in uplifting changesets. Future versions of Autoland may add that capability.
| |
| | |
| == Roadmap ==
| |
| | |
| === Milestone 1 ===
| |
| | |
| A prototype which can be tested end-to-end by pushing to try with a specific syntax.
| |
| | |
| target delivery date: 2014Q4
| |
| | |
| status: in alpha testing, see https://groups.google.com/forum/#!topic/mozilla.tools/sOtaq0P0XHk
| |
| | |
| | |
| <bugzilla>
| |
| {
| |
| "whiteboard": "[autoland M1]"
| |
| }
| |
| </bugzilla>
| |
| | |
| === Milestone 2 ===
| |
| | |
| Full Treeherder integration so that sheriffs can view and approve/reject Autoland pushes that contain repeated test failures.
| |
| | |
| target delivery date: mid 2014Q4
| |
| | |
| | |
| <bugzilla>
| |
| {
| |
| "whiteboard": "[autoland m2]",
| |
| "component": "Autoland"
| |
| }
| |
| </bugzilla>
| |
| | |
| === Future Milestones ===
| |
| | |
| * ReviewBoard Integration
| |
| * Full production deployment, including monitoring, ops documentation, and high-availability
| |
| | |
| == Bugs ==
| |
| Bugs should be filed in Bugzilla under Tree Management:Autoland.
| |
| | |
| <bugzilla>
| |
| {
| |
| "quicksearch": "status:new,assigned,reopened,unconfirmed",
| |
| "product": "Tree Management",
| |
| "component": "Autoland"
| |
| }
| |
| </bugzilla>
| |
| | |
| == Workflow ==
| |
| The proposed workflow for Autoland is as follows:
| |
| | |
| 1 - ReviewBoard or developer submits patch/changeset to try with a specific syntax, maybe --autoland.<br/>
| |
| 1a - If the try run is being requested by ReviewBoard, check that the requester (not the patch author) has commit access level 3, and if not, reject with an appropriate error message. If the try run has been requested directly, this check isn't performed until step 3.
| |
| | |
| 2 - A try run is performed (with a subset of tests if specified by the submitter).<br/>
| |
| 2a - When performed via ReviewBoard, we should encourage targeted try runs in order to conserve resources.<br />
| |
| 2b - When performed via ReviewBoard, we should force the try run against the tip of mozilla-inbound, in order to prevent the case where a changeset may pass a try run against an older gecko commit that the review was created against, but fail when landing on inbound.
| |
| | |
| 3 - An Autoland service somewhere is consuming the pulse stream for the try repo, notices an Autoland job has been submitted to try and starts monitoring it.<br />
| |
| 3a - If the try run was submitted directly (not via ReviewBoard), the Autoland service should check the commit level access of the submitter, and reject the Autoland attempt if the submitter does not have commit access level 3.
| |
| | |
| 4 - When no pending jobs remain, the Autoland service looks for failed jobs; if it finds any it retriggers them.<br />
| |
| 4a - Autoland will determine when all jobs are complete using the buildapi at [https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-central/rev/2d88803a0b9c/is_done]<br />
| |
| 4b - Autoland will determine job status by querying the buildapi<br />
| |
| 4c - Autoland will retrigger failed jobs using the buildapi<br />
| |
| 4d - Initially, Autoland will retrigger failed jobs only once before taking action; this can be tweaked in the future as needed if we want to make it smarter in hunting for frequent intermittent failures introduced by the changeset.
| |
| | |
| 5 - When Autoland service gets results for the retriggered jobs:<br/>
| |
| 5a - If all retriggered jobs pass, it lands the patch in-tree (m-i by default, by developer can request another tree using some TBD try syntax)<br/>
| |
| 5b - If patch no longer cleanly applies, Autoland rejects with a message on the bug<br/>
| |
| 5c - If the retriggered jobs fail, Autoland will annotate the push in Treeherder
| |
| | |
| 6 - Sheriffs periodically monitor the Autoland queue in Treeherder for jobs that need human inspection, and flag them as ok to land or not ok to land, possibly after additional manual retriggers.<br/>
| |
| 6a - If ok to land, autoland lands the patch in-tree. We may coalesce pending autoland commits and land them in batches periodically, like once per hour, ala Bumper Bot.<br/>
| |
| 6b - If not ok to land, Autoland notifies the developer of the problem via bugzilla.
| |
| | |
| == Components ==
| |
| | |
| === Autoland Service [owner: dminor] ===
| |
| This is the primary component of Autoland. This service monitors pulse for autoland jobs submitted to try, monitors such try jobs for completion using the buildapi, queries the buildapi for job status, and retriggers failing jobs. The project repo is located at [1].
| |
| | |
| It also submits comments to Bugzilla when the Autoland status of a try push changes, and lands patches on inbound trees if all success criteria are met. Patches will be landed using the transplant tool [2].
| |
| | |
| [1] https://github.com/dminor/autoland
| |
| | |
| [2] https://github.com/laggyluke/transplant
| |
| | |
| === Treeherder [owner: dminor] ===
| |
| Treeherder will be utilized in the following ways:
| |
| | |
| # Treeherder will expose a special Autoland view of try jobs which need sheriff attention; this view will include a button (available only to privileged users) that will allow a pending Autoland push to be landed or rejected.
| |
| | |
| See {{bug|1058864}}
| |
| | |
| === Releng Buildapi ===
| |
| This api (which exists already) will be used to determine when Autoland try jobs are finished, and will be used to retrigger failing jobs.
| |
| | |
| docs: [[ReleaseEngineering/Applications/BuildAPI]]
| |