Splitting our buildbot master into "pods" might look something like this:

• A controller process
  • Handles incoming changes (from hg)
  • Creates initial optimized build, debug build entries in the Queue
  • Monitors build progress
    • Creates subsequent unit test, talos test entries in the Queue once builds are available
    • Re-queues builds on certain types of failure

• Buildbot masters
  • No hg polling or sendchanges are done any more
  • Only source of requests is by looking in the Queue. When a request that the master can satisfy is found, it can mark that request as taken.

The Queue

A database with the following tables:

• branches
  • id
  • name
  • priority
  • config
    • I think this can just be a big JSON-encoded string, we don't need to run queries on the various parts of the branch config, do we? The config contains things like the hg branch, tinderbox tree names, whether to run unittests, debug builds, etc.

• requests
  • id
  • branch_id
  • type (optimized build, debug build, etc.)
  • revision
  • file_url (required for talos and packaged unittests. combine this with revision somehow?)
  • submit_time
  • complete_time
  • master_id (which master is handled or is currently handling this request)

• request_attempts
  • id
  • request_id
  • build_id (refers to builds.id column in buildbot database)

Open questions

• How do requests get assigned to masters?
  • controller could push the information to the master via a sendchange and/or create an entry in the request_attempts table for that master.
  • master could be querying the requests table directly and updating the requests table when it takes a request

• How does controller notice when builds are done? e.g. when an optimized build is done, talos runs need to be queued.
  • SQL triggers?
  • Poll the DB?
  • Have some kind of 'ping' or sendchange from the buildbot master back to the controller when builds are done.