Release:Release Automation on Mercurial:Plan

Release Automation on Mercurial: Documentation | Plan | Details
This page is no longer relevant to new release automation development and is being kept only for historical purposes
Because of certain things discovered in the development cycle this plan is not entirely up to date. Most things here are still true, though.

Preamble

This page is a proposal for an automated release system for Gecko-based products in Mercurial. It is based off of our existing release process with some tweaks and improvements. Some are fairly invasive but it feels like this is the right time to such changes.

Multiple approaches to this problem have been considered. In the end, I came to the conclusion that because of Bootstrap's and (especially) Tinderbox's tight integration with CVS it would not be worthwhile to modify them to work with both CVS and Mercurial. One route that I considered taking was running l10nverify -> update_verify out of Bootstrap. This was abandoned because I felt it was incompatible with having builds and mars pushed in the releases/ file/directory format. It would end up being a harder and longer job to make Bootstrap support that *and* the old format than it would be to simply leave Bootstrap out of it altogether. As it turns out, the only things that need to be ripped out of Bootstrap are the patcher and update verify config bumping logic. Most of the other things Bootstrap does is call a series of ShellCommands with very little logic involved.

Most steps have something to the effect of 'implement in a BuildFactory with a series of ShellCommands' in the Implementation Notes section. In some cases this will be possible right away. In others it will require either a) a BuildStep subclass, or b) a new Makefile target. Things like uploading to stage.m.o are a perfect example of this. I would like to take this opportunity to go route b) and keep logic like that inside our build system, when it makes sense to do so. I think we should try hard to avoid subclassing. There will be exceptions to this, of course. In particular, log analysis may make a lot of sense to do in a createSummary() method of a ShellCommand subclass.

Release Process

On the left is the automation process for 1.8 and 1.9 releases. On the right, is the proposed automation process for builds off of Mercurial.

'Stage' is the only step that has been removed entirely. It's job is entirely unnecessary because of the changes to the 'Build' step discussed below. More on that and other process changes below.

Tagging/branching

The tree does NOT need to be closed until branching and tagging happens. The tree should be closed for stabilization after the last check-in for a release happens, but after that, normal check-ins may continue. However, the tree DOES need to be closed while the process of branching and tagging happens. This is to avoid complications with possible merge commits.

Deliverables

For all repositories involved in a release:

• A named "release" branch that can be used if a respin is necessary (eg, GECKO191_20080724_RELBRANCH) which contains version bumps if necessary (eg, 3.1a1pre -> 3.1a1).
• The above revision tagged for the current Build # (eg, FIREFOX_3_1a1_BUILD1)
• The above revision tagged as _RELEASE (eg, FIREFOX_3_1a1_RELEASE)

Implementation Notes

• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Source tarball/bundle

This step explicitly does NOT apply to l10n repositories.

Deliverables

• A source tarball (.tar.bz2) of the code used to build a release without version control metadata.
• A Mercurial bundle of the repository/repositories the product is built from. This is inclusive of the very first revision of the repository up to and including the revision tagged for this release.

Implementation Notes

• Generation of the packages should be implemented in a Buildbot BuildFactory as a series of ShellCommands and Mercurial checkout.
• Pushing of the files to stage should be either a straight 'scp' ShellCommand, or a new Makefile target.

Build/Mars

There are two important changes from our existing build process here:

• Generating partial mars right away, instead of with snippets.
• Pushing builds in the same file/dir structure that we release them in.

Deliverables

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

All of the files mentioned above should be named and stored in the format they will be released in. (See ftp://ftp.mozilla.org/pub/firefox/releases/3.0.1 for an example)

Implementation Notes

• The existing logic used for generating builds and complete mars on mozilla-central for nightly builds should be moved to a BuildFactory. Release builds should use this factory to accomplish this task.
• Partial mars should be generated using make_incremental_updates.py.
• Pushing builds/mars to stage should be either a straight 'scp' ShellCommand or a new Makefile target.

l10n repack/mar generation

As with Build, this step generates partial mars and pushes files in their final name/directory structure.

Deliverables

For every locale being shipped:

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

Implementation Notes

TBD

Signing

This step encompasses both the internal EXE/DLL signing of win32 installers AND the installers/packages themselves.

Deliverables

For Windows builds:

• Per locale, an installer with both its internal DLLs/EXEs signed and the installer itself signed (Authenticode)

For each Windows, Linux and Mac locale, and the source tarball:

• a detached gpg signature

Implementation Notes

• sign-release.pl (and maybe other scripts) needs to support builds in the releases/ directory structure and name format.
• Documentation probably needs to be updated.

l10n verification

It is important to note that this step no longer waits on Signing to be completed before starting.

Deliverables

• diff output comparing (l10n vs en-US for the previous release) with (l10n vs en-US for the current release)

Implementation Notes

• l10n verify scripts will need to be modified support release/ style file/directory names.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Snippet generation

As with l10n verification this step no longer waits for Signing to complete before starting.

Deliverables

• An updated and checked-in patcher config file.

For each platform/locale/channel combination:

• A complete update snippet for all previous releases that ponits to the complete mar.
• A partial update snippet for the most recent previous releases that points to the partial mar.
• A partial update snippet for all other previous releases that points to the complete mar.

Implementation Notes

• The logic that updates the patcher config file should be pulled out of Bootstrap and into a stand-alone script.
• Patcher will need to be modified to support release/ style file/directory names.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Update/snippet verification

This step is two distinct tests. One ensures that previous release + applied partial is the same as the new installer, and that previous release + complete update is also the new installer. The other tests aus2 URLs on the 'betatest' channel to ensure all previous builds will be able to download a complete and/or partial mar.

Deliverables

• An updated and checked-in configuration file for each platform.
• Pass/fail notification of both tests mentioned above.
• More detailed information when a failure occurs.

Implementation Notes

• The logic that updates the configuration files should be pulled out of Bootstrap and into a stand-alone script.
• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Bouncer link verification

This test verifies that ensures bouncer is pointing to valid links for every combination of locale/platform/previous version. It must be run after there is good mirror saturation.

Deliverables

• Pass/fail notification
• Full, detailed log to aid analysis in case of failures

Implementation Notes

• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Releasetest/release snippet comparison

This test ensures that the release channel snippets are identical to the releasetest channel snippets.

Deliverables

• Pass/fail notification
• In the case of differences, a list of files that differ.

Implementation Notes

• Should be implemented as a BuildFactory in Buildbot as a series of ShellCommands.