Release:Release Automation on Mercurial:Starting a Release

From MozillaWiki
Jump to navigation Jump to search

<< Documentation

After we get a "go to build" e-mail from Release Management there's some manual steps to be run to prepare for and start a release. This page describes them in detail.

L10N Changesets

Currently we have separate changesets files for Fennec and Firefox. Both are generated from the l10n dashboard
First log in to the dashboard with your LDAP and then follow the instructions below

Creating beta milestones

Sometimes the beta milestones that we need have not been created and we can do so if we have the right permissions:

  • load https://l10n.mozilla.org/shipping/milestones and sign-in
  • click on the link at the bottom of the page: "You may be able to create new milestones."
  • hit "clear aurora" and all other products that you are not releasing (there is an "X" at the beginning of each row)
  • once you see the product(s) that you want to bump you can hit "submit" and the milestones will be created

Fennec

android
  • In the 'Add a multi-locale file for' field put
android-multilocale
  • select "Add"
    • repo: "releases/mozilla-beta" (you MUST use mozilla-beta, even for mozilla-release based releases)
    • branch: "default"
    • path: "mobile/android/locales/maemo-locales"
  • select "Generate it"
  • Copy and paste it into your repo's l10n-changesets_mobile-{version}.json
  • go back to the original page
  • select "Ship it" to preserve the record of what was shipped

Firefox

Note: these instructions also work for Thunderbird.

  • Click "ship" to load up the milestone (eg: Firefox 14 Beta Build 12)
  • This will take you to https://l10n.mozilla.org/shipping/confirm-ship?ms=fx14_beta_b12
  • Open in a new tab the "l10n-changesets" URL (don't touch any of the other fields) record the URL on the build notes (e.g. https://l10n.mozilla.org/shipping/l10n-changesets?ms=fx14_beta_b12)
    • all other fields are only relevant to Fennec milestones (bad UI used for both type of products)
  • copy and paste the list into your l10n-changesets_mozilla-{version}
  • go back to the original page
  • select "Ship it"
    • if you forget the URL, you can recreate the URL by judicious modification of the URL from the prior build notes

Finding Build Master

Only specific build masters should be used for release builds, and these are allocated to specific branches. Find the correct choice for your builds from the production-masters.json file. (This script can parse the json and output what you need.)

Clobbering

Mark a clobberer for "Any master", $branch, "Any builder" with clobberer. This will instruct any slave that did the last release on $branch to clobberer their release directories. Ideally, this should be done hours or more before the release starts, so that cleanup occurs before the release job starts. Clobbering isn't mandatory if your release config has base_clobber_url pointing to always_clobber.php.

If you are doing a test release in staging there is additional setup work to do, and the clobberer URL changes.

Locking slaves

Because releases happen on only one master and slaves are scattered across all of them, it's necessary to make sure you have enough slaves to complete the release in a timely manner prior to starting. Generally, you need at least 3 or 4 for each platform (win32, win64, macosx64, linux, linux64 you build on, but 6 is necessary to fully parallelize l10n repacks for minimum time elapsed. Use the Buildbot interface for each build platform (eg, http://buildbot-master12.build.scl1.mozilla.com:8001/builders/release-mozilla-release-win32_build) and verify that you have them. If you don't have enough use Slavealloc to lock additional slaves to your master. If you don't know how to do this, ask someone. If you the slaves you reallocate in slavealloc aren't currently connected to the master that you'd like them to be, you'll need to reboot them after you tie them to the new master.

Finding which platforms are involved in a given release

Convenient URL's for finding which platforms are involved in a given release type:

Minimum Requirements

It may take a while to corral all the slaves needed. The minimum required to start a release are:

  • 3 linux,
  • 2 of everything else (2nd for xul-runner, so only need 1 for thunderbird)

Setting reserved slaves

Aside from making sure you have enough slaves attached to the master, you also need to make sure that those slaves aren't busy doing non-release jobs. For this, reserved slaves is used. It improves the odds of having slaves available when the release starts, which helps reduce end to end time. Reserved slaves is a simple file on the master that's read on a timer by the master. To set it, something like the following should be done a few hours ahead of the release starting (when possible): 

cd /builds/buildbot/build1/master
echo 8 > reserved_slaves

Starting the automation

Generally, the workflow for kicking off Release Automation is as follows:

  • For build1 only, create a symlink to the candidates directory (until bug 725839 is resolved). For example:
# ffxbld or tbirdbld@stage
version=13.0b6
product=thunderbird
cd /pub/mozilla.org/$product/nightly/
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates
  • Land configuration file updates. Double-land on the production branch if you aren't merging default -> production.
  • Tag buildbotcustom (branch production-0.8), buildbot-configs (branch production), tools (branch default) with the correct _RELEASE and _BUILDn tags. For example, FENNEC_14_0b6_RELEASE and FENNEC_14_0b6_BUILD1. Multiple production releases require tags for both products.
  • Update & reconfig the master doing the release (make update and make reconfig)
  • Run release sanity. For example:
# Combined release
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
    -V 6.0b2 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py \
    --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001
# Firefox-only release
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum \
 -V 6.0b2 --branch mozilla-beta --build-number 1 \
 --release-config release-firefox-mozilla-beta.py --products firefox  \
 --dryrun localhost:9001
# Thunderbird release:
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u hwine \
 -V 14.0b5 --build-number 1 --branch comm-beta \
 --release-config release-thunderbird-comm-beta.py --product thunderbird \
 --dryrun localhost:9001
  • Start the automation by running the sanity script again, without --dryrun.

If you're working in staging you must make sure to pass in the correct staging release config (staging_release-firefox-<branch name>.py)

Retrieved from "https://wiki.mozilla.org/index.php?title=Release:Release_Automation_on_Mercurial:Starting_a_Release&oldid=459779"