Release:Release Automation on Mercurial:Plan

Preamble

This page is a proposal for an automated release system for Gecko-based products in Mercurial. It is based off of our existing release process with some tweaks and improvements. Some are fairly invasive but it feels like this is the right time to such changes.

Multiple approaches to this problem have been considered. In the end, I came to the conclusion that because of Bootstrap's and (especially) Tinderbox's tight integration with CVS it would not be worthwhile to modify them to work with both CVS and Mercurial. One route that I considered taking was running l10nverify -> update_verify out of Bootstrap. This was abandoned because I felt it was incompatible with having builds and mars pushed in the releases/ file/directory format. It would end up being a harder and longer job to make Bootstrap support that *and* the old format than it would be to simply leave Bootstrap out of it altogether. As it turns out, the only things that need to be ripped out of Bootstrap are the patcher and update verify config bumping logic. Most of the other things Bootstrap does is call a series of ShellCommands very little logic involved.

Release Process

On the left is the automation process for 1.8 and 1.9 releases. On the right, is the proposed automation process for builds off of Mercurial.

The reasoning behind these changes will be discussed in the sections below.

Tagging/branching

Deliverables

For all repositories involved in a release:

• A named "release" branch that can be used if a respin is necessary (eg, GECKO191_20080724_RELBRANCH) which contains version bumps if necessary (eg, 3.1a1pre -> 3.1a1).
• The above revision tagged for the current Build # (eg, FIREFOX_3_1a1_BUILD1)
• The above revision tagged as _RELEASE (eg, FIREFOX_3_1a1_RELEASE)

Implementation Notes

• The tree should be closed for the duration of this step to avoid any possibility of merge commits.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Source tarball/bundle

This step explicitly does NOT apply to l10n repositories.

Deliverables

• A source tarball (.tar.bz2) of the code used to build a release without version control metadata.
• A Mercurial bundle of the repository/repositories the product is built from. This is inclusive of the very first revision of the repository up to and including the revision tagged for this release.

Implementation Notes

• Generation of the packages should be implemented in a Buildbot BuildFactory as a series of ShellCommands and Mercurial checkout.
• Pushing of the files to stage should be either a straight 'scp' ShellCommand, or a new Makefile target.

Build/Mars

There are two important changes from our existing build process here:

• Generating partial mars right away, instead of with snippets.
• Pushing builds in the same file/dir structure that we release them in.

Deliverables

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

All of the files mentioned above should be named and stored in the format they will be released in. (See ftp://ftp.mozilla.org/pub/firefox/releases/3.0.1 for an example)

Implementation Notes

• The existing logic used for generating builds and complete mars on mozilla-central for nightly builds should be moved to a BuildFactory. Release builds should use this factory to accomplish this task.
• Partial mars should be generated using make_incremental_updates.py.
• Pushing builds/mars to stage should be either a straight 'scp' ShellCommand or a new Makefile target.

l10n repack/mar generation

As with Build, this step generates partial mars and pushes files in their final name/directory structure.

Deliverables

For every locale being shipped:

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

Implementation Notes

TBD

Signing

This step encompasses both the internal EXE/DLL signing of win32 installers AND the installers/packages themselves.

Deliverables

For Windows builds:

• Per locale, an installer with both its internal DLLs/EXEs signed and the installer itself signed.

For Linux and Mac builds:

• Per locale, a signed package.

Implementation Notes

• sign-release.pl (and maybe other scripts) needs to support builds in the releases/ directory structure and name format.
• Documentation probably needs to be updated.

l10n verification

It is important to note that this step no longer waits on Signing to be completed before starting.

Deliverables

• diff output comparing (l10n vs en-US for the previous release) with (l10n vs en-US for the current release)

Implementation Notes

• l10n verify scripts will need to be modified support release/ style file/directory names.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Snippet generation

Deliverables

• An updated and checked-in patcher config file.

For each platform/locale/channel combination:

• A complete update snippet for all previous releases that ponits to the complete mar.
• A partial update snippet for the most recent previous releases that points to the partial mar.
• A partial update snippet for all other previous releases that points to the complete mar.

Implementation Notes

• The logic that updates the patcher config file should be pulled out of Bootstrap and into a stand-alone script.
• Patcher will need to be modified to support release/ style file/directory names.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Update/snippet verification

This step is two distinct tests. One ensures that updating from a partial mar has the same end result as updating from a complete mar. The other tests aus2 URLs on the 'betatest' channel to ensure all previous builds will be able to download a complete and/or partial mar.

Deliverables

• An updated and checked-in configuration file for each platform.
• Pass/fail notification of both tests mentioned above.
• More detailed information when a failure occurs.

Implementation Notes

• The logic that updates the configuration files should be pulled out of Bootstrap and into a stand-alone script.
• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Bouncer link verification

This test verifies that ensures bouncer is pointing to valid links for every combination of locale/platform/previous version. It must be run after there is good mirror saturation.

Deliverables

• Pass/fail notification
• Full, detailed log to aid analysis in case of failures

Implementation Notes

• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Releasetest/release snippet comparison

This test ensures that the release channel snippets are identical to the releasetest channel snippets.

Deliverables

• Pass/fail notification
• In the case of differences, a list of files that differ.

Implementation Notes

• Should be implemented as a BuildFactory in Buildbot as a series of ShellCommands.