Build system design

This document contains design specifications for the BYOB build system.

We're not doing anything horribly complex here, so rather than reinvent the wheel, I recommend we go with a simple buildbot master/slave setup with custom hg pollers to trigger repacks.

Why use buildbot?

• open source with an active development community
• leverage existing institutional knowledge in the RelEng team

Master setup

The buildbot master can run either Mac or Linux.

Based on RelEng experience with buildbot masters, a physical machine is recommended *if* we project an order-of-magnitude increase in BYOB submissions. At present, we're unlikely to see too much load (comparable with our core build systems), so a VM would likely suffice.

Slave setup

Due to Mac packaging requirements, Mac slaves are required. These can be physical machines (recycled minis perhaps?) or VMs if the OS X VMs prove performant enough.

If we are going to sign our own BYOB builds, we will also need a Windows slave. Again, this slave can be a physical machine or a VM.

If we plan to offer BYOB for mobile, we'll need to install the appropriate Debian packaging tools for Mac.

Depending on projected demand for the BYOB service, 3-5 slaves would be a good initial investment.

Repack hosting and storage

We'll need a file server where we can stage the in-progress (unsigned) and completed repacks. I defer any decisions here to IT.

Required customizations

Assumptions:

• configurations will be stored in a version control repository (Mercurial)
• Windows repacks need to be signed, so the repack process for Windows builds will differ slightly than for other platforms. Windows repacks will initially end up in a temporary dir pending signing.

We can setup a number of different custom triggers/hg pollers depending on how we want to generate builds. Here's a list of the triggers I think we need:

• build per check-in: triggers all builds changed by the check-in. Builds are broken up by repack so that they can be parallelized across the build slaves. We'll have a suitable idle timer in place to catch "oops, i missed one thing"-type errors.
• one-off/single builds: can be triggered from the buildbot master directly, but we could also hook up a sendchange request to the webtool to accomplish the same thing.
• re-generate all active BYOB repacks: this will likely be a custom sendchange that will generate a job per repack. This will be useful for security releases, and can be hooked up to the webtool.
• signing poller: looks for builds in an incoming directory, signs them, puts them into the final location.