User:AxelHecht/Build

I'm trying to get my act together on what I want from the build infrastructure. So here's my assorted brain dump.

Overview

We start off with

• The /cvsroot repository and changes for this
• The /l10n repository and changes for this

In addition, there are some human requests, namely

• Nightly builds
• Clobbers
• Kicks

What *I* want are

• Localized builds, working ones, even
• Localization statistics, like, how long does it take which locale to update
• Test results, say, Minotaur and others

Localizers may have a different mileage.

Whether just nightly builds are OK, or whether we should have full blown l10n builds is something that's not really clear to me, and I'm not sure what affects the decision. Disk space sounds like one.

One misty London night is the time between the last nightly and a current check-in in the l10n repository. It's not really clear what the target of that check-in is, and what's the right platform to test it against. It could be the last nightly, it could be the upcoming nightly, or neither. I guess we'll need to get experience. Trying to get many tests run as source tests might help, too.

The first chunk of my thinking goes into what to do with en-US build and how to get l10n for those. This is prominently involving nightlies.

New en-US Build

Let's see some artwork:

This process is either time triggered or otherwise, but it's basically a "build the tree at a timestamp". We start off with the en-US and the l10n reps, do whatever needs to be done for en-US, while doing that do l10n-merge on an l10n checkout, and then distribute the localized merged sources to platform specific repack slaves. The l10n tarball is really just there to efficiently transport the trees. I haven't figured out what to do with the result of automatic tests, neither for en-US nor for l10n, but we should probably do something 'clever' for that.

Continuous l10n builds

There is some mind-bending involved to find out which localization from which branch should be built against which branch. The fact that L10n:Incubator builds the l10n trunk against the en-US branch for a few locales doesn't help here, but it helps the new localizers a bunch.

Here's the underlying flow of information implemented on L10n:Buildbot and currently under review in bug 393127.

To make things only mildly confusing, I left out the multiplicity of messages in the diagram. And I only depicted one tree.

Concepts

Let's write down some of the key concepts.

Trees

A single tree stores the information of the branch combination,

• en-US branch (i.e. MOZILLA_1_8_BRANCH for incubator)
• l10n branch (i.e. HEAD for incubator)

the list of applications or the list of languages for each requested app. In the first case, the list of built languages is retrieved from mozilla/$(app)/locales/all-locales. That means, a single tree can be building selected languages for fx and tb (and suite), or all locales for a set of selected apps.

• apps (list of applications) or locales (hash of locales per app)

Trees are identified by a name.

Dispatchers

The l10n scheduler defers to helper objects to find out what to build when. These helper objects are called dispatcher. For each tree that the scheduler handles, there is an L10nDispatcher and an EnDispatcher. Whenever the buildbot ChangeMaster gets a change notification for the bonsaipollers, it's feeding those changes to the scheduler, which forwards every change to every dispatcher. The dispatcher decides based on whether it's and l10n dispatcher or not, and the tree configuration data it has, on whether it should handle this change. If it does, it's finding out the affected applications, and the affected localizations. For the L10nDispatcher, the affected localizations are those that changed, for the EnDispatcher, all enabled localizations for that tree and the affected products is kicked off.

Each dispatcher then calls into Scheduler.queueBuild with the language, the tree, the app, and the requested builders. The scheduler then queues those builds, possibly merging duplicate requests, and hands them off to the builders.

Technical detail, the scheduler uses a subclassed build request, which is not mergeable. That way, we can count on the builder to distribute the requests, one by one, to all its slaves.

Builders

Builders are set up per build process. They are independent of the application they build, but they do depend on the platform. The latter is due to the fact that the slaves are bound to platforms, and bound to builders.

Which application is repackaged (or langpacked) for which language is determined by properties on the build, which the Build class (buildbotcustom.l10n.Build) pulls off of the schedulers build queue.