BugzillaAutoLanding

Project Description

This project will create and deploy a set of tools that can grab patches from a bug in Bugzilla, land them on try, poll for the results of that push and if the results meet criteria auto-land the patch(es) on trunk, poll for results, comment back to the bug with trunk landing results - backing out if the results do not meet criteria for landing on trunk.

Goals

• A set of smaller tools that each have the ability to be part of a larger script that runs "autolanding" to try & trunk.
• Tools can be used in command line
• Tools to control the process - lots of toggles to go back to manual sheriffing (Global KillSwitch)
• Using our build/test resources wisely and not increasing the load so much that try is unusable for developers to work on their patches prior to automated/assisted landings
• Streamlining as much as possible the try server and trunk landing process without removing the human interaction with results.

Non-Goals

• Replacing humans in the landing process
• Handling performance regressions
• Providing the best possible UI
• Going to extreme lengths to support auto-landing patches which do not follow the rules which the tool requires

People

• Lukas Blakk
• Marc Jessome

Bugs

• Tracking bug: bug 657828
  • bug 430942 -- lsblakk - for creating the SchedulerDB Poller
  • bug 657832 -- mjessome - Allow for automatically pushing to $branch when [autoland-$branch] is in whiteboard tags

Designing the System

A simple survey on try usage was created and advertised to find out how developers currently use try and what they think about the autolanding workflow. We got 52 responses. This helped catch some things we missed like paying attention to LDAP authentication before pushing something from Bugzilla, and it also provided new observations on try workflow. Many developers state that they would find landing to try via Bugzilla to be 'more work' than what is currently offered with push-to-try (using try syntax). We are taking that into account in our design and are adjusting the goal to be primarily for the purposes of landing to trunk and not just getting try results posted to the bug. That last part will be handled more through try syntax where if a bug is specified, you can post the results to the bug and turn off email notification if desired.

API

Architecture

REST Interface

List of request methods, urls, and parameters:

POST /bugs/{bugID} -> insert bug

POST /bugs/{bugID}/{patchSetID} -> insert an auto-land patchset

GET /bugs/{bugID} -> list the patchSet(s) for a bug (to support multiple autolanding attempts on the same bug)

GET /bugs/{bugID}/{patchSetID} -> information about a patchSet

GET /bugs/{bugID}/{patchSetID}/{patchID} -> information about a particular patch (link to bugzilla attachment?)

PUT /bugs/{bugID}/{patchSetID}/{patchID}(s) -> update this bug - add patchID(s) to a bug's patchSet assigned by DB on insert

• priority
• patchID(s)

PUT /bugs/{bugID}/{patchSetID} -> update the pushID

• tryresult/branchresult from SchedulerDBPoller
• pushresult from HgPusher

DELETE /bugs/{bugID} -> cancel all queued pushes for this bug and move to completed db table?

DELETE /bugs/{bugID}/{patchSetID} -> cancel a patchSet from the list of patchSets queued for a bug

DELETE /bugs/{bugID}/{patchSetID}/{patchID} -> remove a patch from the queued push

Object Definitions

Bugs:

• bugID
• patchSets

PatchSets:

• patchSetID
• patches
• revision
• pushresult
• tryresult
• branchresult
• final_destination
• comments
• priority

Patches:

• patchID
• author
• reviewer
• approver

Comments:

• commentID
• message

Project Timeline

Component Name	Bug(s)	Assigned To	Description	Start Date (est.)	Completion Date (est.)	On Track
HgPusher	bug 657832	Marc	Details	Monday May 23rd	Monday June 20th	Yes
SchedulerDBPoller	bug 430942	Lukas	Details	Monday May 23rd	Monday June 20th	Yes
BugCommenter	bug #	tbd	Details	tbd	tbd	TBD
AutolandDB	bug #	tbd	Details	tbd	tbd	TBD
MessageQueue	bug #	tbd	Details	tbd	tbd	TBD

Component Descriptions and Implementation Notes

HgPusher

Accepts a branch, patch id(s) and can clone the branch, apply the patches and report back with results of push (success == revision or FAIL). Also can handle special casing to do a backout on a branch.

SchedulerDBPoller

Regularly polls the scheduler DB (on a timer) and checks for completed buildruns for any actively monitored branches. There should be a list to check against for what branches need to be watched. The incomplete runs from the last N units of time are kept by revision in a local cache file to check for completion. When an observed branch has a completed buildrun the SchedulerDBPoller can check two things:

• if try syntax is present (and branch is try) check for a --post-to-bug flag and trigger BugCommenter if flag and bug number(s) are present
• run that revision against the AutolandDB to see if that revision was triggered by landing automation. If yes, then PUT the results & trigger the BugCommenter otherwise ditch the completed revision

BugCommenter

• When called with a bug number and a comment, posts to the bug and returns the commentID as well as a result (SUCCESS/FAIL) -- should handle a few retries in case of network issues

MessageQueue

To-Do - Read up on RabbitMQ and Message Queue

• Listens to messages from BugzillaScraper (or Pulse Events?)
• Broadcasts information to HgPusher, BugCommenter, AutolandDB

AutolandDB

Keep track of the state of an autoland-triggered push

Notes

• Set of tools which automate each step of this work and we can 'on-demand' turn off any of the tools
• Sheriff needs to be able to turn off the auto-landing altogether
• No limit in workflow -- sheriff should be able to override the queue to auto-land a priority patch
  • can have oranges, but sheriff could 'force' the landing anyway
• We need to have stages:
  • (TBD): how to deal with bugzilla
  • HgPusher: deals with the hg stuff (merging patches, grabbing patches from bugs, what happens if any steps fail)
  • SchedulerDB Poller: deals with a way of getting results from try/m-c to make sure we have the data on whether to proceed and/or leave comment in the bug regarding outcomes
  • (fourth, optional/desired/future) how to monitor perf results and get an idea of when it's ok to push or flag on those results
• Be strict in accepting things from humans
  • Rules for automated landings
  • If they forget a rule, they get a comment - "Step X failed (reason)"
  • For bugs with multiple patches - to land all together, look at non-obsolete patches which have been reviewed in alphabetical order
  • Common for devs to name things "part 1", "part 2"
  • Don't need to be lenient toward human mistakes
  • Need correct descriptions, author information, header of patch -- if it doesn't include a header, try syntax
  • Look at "checkin-needed" box, grab all non-obsolete, and if they have the right message then push all those - if all those steps succeed then you get the try push otherwise fail with comment -- clear "checkin-needed" from the bug
• Is the bug the right place? It's the public record for the issue, so yes - emailing the assignee takes the information away from the record
• Specialized tool for patch queues would be nice (not in the scope of this project!)
• Try syntax presence == try on try_repo but NEVER auto-land on trunk
• Developers will be encouraged to watch perf results on trunk, since try perf is not really useful information
• Bot has to watch m-c after it lands something there, results come back to the bug
  • Merge tracking for commits (what was auto-landed and what was not)
  • Merging between two auto-landed pushes
  • We have to watch these to know when all the results are in, and what was successful/not (scheduler db has this information for us to get)
• Autoland Message Queue - consumes information from BugzillaScraper (pulse?) as event triggerer for HgPusher, BugCommenter

Security

• Must ensure that the patch is attached by someone with L1 hg access, so that we are auto-landing patches from authors with the same level of security as current push-to-try
  • Why L1? This is about pushing to mozilla-central, so I think we should check for L3 access.
  • L1 to push to try, then reviewer should have L3 for autolanding to trunk