Release:Release Automation on Mercurial:Plan

Release Automation on Mercurial

This page is a proposal for an automated release system for Gecko-based products in Mercurial. The process is based around our existing automation (Bootstrap) and manual processes around it.

Overview

[talk about not using custom buildbot steps]

Release Process

The automated process we follow for 1.8 and 1.9 builds looks like this:

This proposal suggests we alter the process to work as follows:

The reasoning behind these changes will be discussed in the sections below.

Tagging/branching

Deliverables

For all repositories involved in a release:

• A named "release" branch that can be used if a respin is necessary (eg, GECKO191_20080724_RELBRANCH) which contains version bumps if necessary (eg, 3.1a1pre -> 3.1a1).
• The above revision tagged for the current Build # (eg, FIREFOX_3_1a1_BUILD1)
• The above revision tagged as _RELEASE (eg, FIREFOX_3_1a1_RELEASE)

Implementation Notes

• The tree should be closed for the duration of this step to avoid any possibility of merge commits.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Source tarball/bundle

This step explicitly does NOT apply to l10n repositories.

Deliverables

• A source tarball (.tar.bz2) of the code used to build a release without version control metadata.
• A Mercurial bundle of the repository/repositories the product is built from. This is inclusive of the very first revision of the repository up to and including the revision tagged for this release.

Implementation Notes

• Generation of the packages should be implemented in a Buildbot BuildFactory as a series of ShellCommands and Mercurial checkout.
• Pushing of the files to stage should be either a straight 'scp' ShellCommand, or a new Makefile target.

Build/Mars

There are two important changes from our existing build process here:

• Generating partial mars right away, instead of with snippets.
• Pushing builds in the same file/dir structure that we release them in.

Deliverables

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

All of the files mentioned above should be named and stored in the format they will be released in. (See ftp://ftp.mozilla.org/pub/firefox/releases/3.0.1 for an example)

Implementation Notes

• The existing logic used for generating builds and complete mars on mozilla-central for nightly builds should be moved to a BuildFactory. Release builds should use this factory to accomplish this task.
• Partial mars should be generated using make_incremental_updates.py.
• Pushing builds/mars to stage should be either a staright 'scp' ShellCommand or a new Makefile target.

l10n repack/mar generation

As with Build, this step generates partial mars and pushes files in their final name/directory structure.

Deliverables

For every locale being shipped:

• Per platform, the appropriate package or installer.
• Per platform, a complete mar file.
• Per platform, a partial mar file against the previous release.

Implementation Notes

TBD

Signing

This step encompasses both the internal EXE/DLL signing of win32 installers AND the installers/packages themselves.

Deliverables

For Windows builds:

• Per locale, an installer with both its internal DLLs/EXEs signed and the installer itself signed.

For Linux and Mac builds:

• Per locale, a signed package.

Implementation Notes

• sign-release.pl (and maybe other scripts) needs to support builds in the releases/ directory structure and name format.
• Documentation probably needs to be updated.

l10n verification

Deliverables

• diff output comparing (l10n vs en-US for the previous release) with (l10n vs en-US for the current release)

Implementation Notes

• l10n verify scripts will need to be modified support release/ style naming.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Snippet generation

Deliverables

• An updated and checked-in patcher config file.

For each platform/locale/channel combination:

• A complete update snippet for all previous releases that ponits to the complete mar.
• A partial update snippet for the most recent previous releases that points to the partial mar.
• A partial update snippet for all other previous releases that points to the complete mar.

Implementation Notes

• The logic that updates the patcher config file should be pulled out of Bootstrap and into a stand-alone script.
• Should be implemented in a Buildbot BuildFactory as a series of ShellCommands.

Update/snippet verification

This step is two distinct tests. One ensures that updating from a partial mar has the same end result as updating from a complete mar. The other tests aus2 URLs on the 'betatest' channel to ensure all previous builds will be able to download a complete and/or partial mar.

Deliverables

• An updated and checked-in configuration file for each platform.
• Pass/fail notification of both tests mentioned above.
• More detailed information when a failure occurs.

Implementation Notes

• The logic that updates the configuration files should be pulled out of Bootstrap and into a stand-alone script.
• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Bouncer link verification

This test verifies that ensures bouncer is pointing to valid links for every combination of locale/platform/previous version. It must be run after there is good mirror saturation.

Deliverables

• Pass/fail notification
• Full, detailed log to aid analysis in case of failures

Implementation Notes

• Should be implemented in a BuildFactory in Buildbot as a series of ShellCommands.

Releasetest/release snippet comparison

This test ensures that the release channel snippets are identical to the releasetest channel snippets.

Deliverables

• Pass/fail notification
• In the case of differences, a list of files that differ.

Implementation Notes

• Should be implemented as a BuildFactory in Buildbot as a series of ShellCommands.