Release:Release Automation on Mercurial:Starting a Release

<< Documentation

After we get a "go to build" e-mail from Release Management there's some manual steps to be run to prepare for and start a release. This page describes them in detail.

L10N Changesets

Currently we have separate changesets files for Fennec and Firefox. Both are generated from the l10n dashboard
First log in to the dashboard with your LDAP and then follow the instructions below

Create milestones

Sometimes the milestones have not been created and we can do so:

• load https://l10n.mozilla.org/shipping/milestones and sign-in
• click on the link at the bottom of the page: "You may be able to create new milestones."
• click the checkbox(es) for the product(s) you want to bump.
  • For betas, make sure the beta # in both fields matches the one you're doing (eg "Beta Build 4" == "Beta 4")
  • For final releases clear the first field and remove "_beta_b#" from the second field.
• hit "submit" and the milestones will be created.

Ship the milestone

Note: when clicking "Ship It" you may be notified that there are pending sign-offs for one or more locales. It is OK to proceed for betas. For final releases we leave the decision with Release Management.

Fennec

• Click the "ship" button to load up the milestone (eg: Fennec 16 Beta Build 3)
• It will tell you if there are pending sign offs. Click "ship it" if you meet on of the two conditions
  • For betas, carry forward regardless if there are pending sign offs
  • For final releases, carry forward *only* if there are no pending sign offs. Ask RelMan if there are
• You will land in a page like this e.g. (https://l10n.mozilla.org/shipping/about-milestone/fennec16_beta_b3)
• Click "shipping tools" at the top
• Platforms should read "android"; multi-locale should read "android-multilocale".
• Click "Add" and check these values
  • repo: "releases/mozilla-beta" (you MUST use mozilla-beta, even for mozilla-release based releases)
  • branch: "default"
  • path: "mobile/android/locales/maemo-locales"
• Click "l10n-changesets.json"
• Copy and paste it into your repo's l10n-changesets_mobile-{version}.json

Firefox or Thunderbird

• Click "ship" to load up the milestone (eg: Firefox 16 Beta Build 3)
• It will tell you if there are pending sign offs. Click "ship it" if you meet on of the two conditions
  • For betas, carry forward regardless if there are pending sign offs
  • For final releases, carry forward *only* if there are no pending sign offs. Ask RelMan if there are
• This will take you to https://l10n.mozilla.org/shipping/about-milestone/fx16_beta_b3
• Click "shipping tools" at the top
• Click "l10n-changesets"
• Copy and paste the list into your changesets file (eg: l10n-changesets_mozilla-{version})

Clobbering

Mark a clobberer for "Any master", $branch, "Any builder" with clobberer. This will instruct any slave that did the last release on $branch to clobberer their release directories. Ideally, this should be done hours or more before the release starts, so that cleanup occurs before the release job starts. Clobbering isn't mandatory if your release config has base_clobber_url pointing to always_clobber.php.

If you are doing a test release in staging there is additional setup work to do, and the clobberer URL changes.

Setting reserved slaves

Aside from making sure you have enough slaves attached to the master, you also need to make sure that those slaves aren't busy doing non-release jobs. For this, reserved slaves is used. It improves the odds of having slaves available when the release starts, which helps reduce end to end time. Reserved slaves is a simple file on the master that's read on a timer by the master. To set it, something like the following should be done a few hours ahead of the release starting (when possible):

cd /builds/buildbot/build1/master
echo 8 > reserved_slaves

Post a Patch for Review

Note: Firefox releases also include Fennec unless otherwise specified.

For a Firefox beta, you'll need to modify:

mozilla/l10n-changesets_mobile-beta.json
mozilla/l10n-changesets_mozilla-beta
mozilla/release-fennec-mozilla-beta.py
mozilla/release-firefox-mozilla-beta.py

Starting the automation

Generally, the workflow for kicking off Release Automation is as follows:

• Land configuration file updates. Double-land on the production branch if you aren't merging default -> production.
• Tag buildbotcustom (branch production-0.8), buildbot-configs (branch production), tools (branch default) with the correct _RELEASE and _BUILDn tags. For example, FENNEC_14_0b6_RELEASE and FENNEC_14_0b6_BUILD1. Multiple production releases require tags for both products.
• Update & reconfig the master doing the release (make update and make reconfig)
• Run release sanity. For example:

# Combined release
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
    -V 6.0b2 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001
# Firefox-only release
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
 -V 6.0b2 --branch mozilla-beta --build-number 1 \
 --release-config release-firefox-mozilla-beta.py --products firefox  \
 --dryrun localhost:9001
# Thunderbird release:
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u hwine \
 -V 14.0b5 --build-number 1 --branch comm-beta \
 --release-config release-thunderbird-comm-beta.py --product thunderbird \
 --dryrun localhost:9001

• Start the automation by running the sanity script again, without --dryrun.

If you're working in staging you must make sure to pass in the correct staging release config (staging_release-firefox-<branch name>.py)