Why?

• Buildbot is supported by a vibrant, active community, and is used by lots of other projects.
• Buildbot itself goes through a continuous integration process, and requires unit tests for code changes.
• Buildbot gives developers more direct control and insight over the currently running builds with features such as stopping/starting builds from the web interface and real-time log output.
• Buildbot lets us use machines more effectively by building in parallel and only building when a checkin happens. This reduces cycle time, uses less overall resources, and gives us a build for each and every changeset. Contrast with our current method of standalone machines which build as often as possible, regardless of whether there's been a change or not.

Shorter term

It's probably fastest and safest to keep Tinderbox server in place, with Buildbot reporting to it, at least until we get the Buildbot/Tinderbox server parity issues figured out.

Use existing release automation

On the client side, we have been working on automating the release process using Bootstrap driven by Buildbot; making this system work for Nightly(Clobber) and Depend builds should be fairly trivial.

Bootstrap's "Build" and "Repack" (l10n) steps currently just call into Tinderbox client, but we mitigate a lot of Tinderbox's problems. We should eventually take it out, but it will be much easier when it's encapsulated into these "steps" instead of our current situation (Tinderbox does everything).

Replace what we have, get the following "for free":

• only build on checkin for Depend
• produce full l10n set with nightly
• better logging/error checking

What we need to get there:

1. buildbot master set up
2. buildbot slaves on Mozilla1.8 and Firefox tinderboxes
3. nightly timer scheduler
4. bonsai change scheduler

We can use the release automation systems for this ^ The above is done on staging environment, results publishing to Mozilla1.8 Staging Tinderbox tree

Depend build scheduler does:

• Build

Nightly build scheduler does:

• Build
• Repack
• Update

We need #1 and #2 done for release automation anyway, #3 and #4 are trivial.

Longer Term

Drive build system from Buildbot

Long-term we can have Buildbot drive the process at a lower level.

The process could be something like:

4 builders total:

• Source (checks out using timestamp, creates tarball)
• (Linux/Mac/Windows)-(Clobber/Depend)

Clobber builds are triggered by the nightly scheduler (or manually) and do the following:

1. Source builder creates and uploads source tarball
2. (Linux/Mac/Windows)-Clobber builders
   1. download source, unpack
   2. configure
   3. make package
   4. upload package
   5. build symbols
   6. upload symbols
   7. create full MAR
   8. upload full MAR

Depend builds are triggered by checkins (e.g. using BonsaiPoller) (or manually) and do the following;

1. (Linux/Mac/Windows)-Depend builders
   1. update existing source checkout (using client.mk)
   2. configure
   3. make package
   4. upload package

Need the following to have this well-supported in buildbot:

• Source() class for client.mk-style scripted checkouts
• some way to do clobber/depend in the same builder - perhaps a custom BuildFactory