Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

• bug 530808 - support locales in beta (mn, sw were in the changesets on the dashboard)
• bug 734644 - hg.m.o time out during Firefox 11.0 build
• bug 734645 - l10n verification times out unpacking files
• Slow (4.5h) linux64 build, no ix slaves attached.
• esr10 was on the same master, aki moved it to bm08

Build Engineers

rail - Tracking bug: bug 718778

Signed-off Revision(s)

• Build 1: c081ebf13261
• Build 2: 398407cca90c

L10N changesets

Instructions on how to get them

• Firefox: l10n-changesets
• Fennec: N/A

Tags

Build #	Branch, Tags	Changeset
1	GECKO110_2012030912_RELBRANCH FIREFOX_11_0_BUILD1 FIREFOX_11_0_RELEASE	72e3dc061978
2	GECKO110_2012031218_RELBRANCH FIREFOX_11_0_BUILD2 FIREFOX_11_0_RELEASE	b967d9c07377

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120309131123	mv-moz2-linux-ix-slave09	1 hrs, 4 mins, 46 secs
	Linux64		moz2-linux64-slave08	4 hrs, 39 mins, 53 secs
	Mac		moz2-darwin10-slave45	3 hrs, 13 mins, 56 secs
	Windows		mw32-ix-slave13	3 hrs, 7 mins, 46 secs
2	Linux	20120312181643	mv-moz2-linux-ix-slave06	1 hrs, 16 mins, 53 secs
	Linux64		linux64-ix-slave05	1 hrs, 14 mins, 38 secs
	Mac		moz2-darwin10-slave50	3 hrs, 25 mins, 21 secs
	Windows		mw32-ix-slave13	3 hrs, 25 mins, 26 secs

Notes

The release is Firefox only, Fennec is supposed to be released off mozilla-esr10

Build 1

Syncing repos

03-08-2012: DONE mozilla-beta -> mozilla-release & l10n repos sync Note - new locale, lij, didn't have anything in the l10n/mozilla-release/lij repo so I had to push the beta repo to it:

# in lij.beta
hg push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/l10n/mozilla-release/lij

Then continued the script with a subset of locales

• Sanity check:

diff -Naur --exclude='.hg*' mozilla-beta mozilla-release > beta-release.diff

• Inspected beta-release.diff. The differences were expected (shipped-locales without mn, sw and android mozconfig changes landed after migration)

Preparing to start Automation

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
• For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/
mkdir ../candidates/11.0-candidates
ln -s ../candidates/11.0-candidates 11.0-candidates

• Reserve slaves

echo "10"> master/reserved_slaves_bm08-build1

• Update l10n changesets for desktop
• Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags (FIREFOX_11_0_RELEASE FIREFOX_11_0_BUILD1)
• update and reconfigure the master
• start automation

# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --dryrun localhost:9001

• Failed because beta-only locales was listed on the l10n dashboard:

2012-03-09 11:48:54,898 : ERROR :       locale mn missing in config
2012-03-09 11:48:54,898 : ERROR :       locale sw missing in config
2012-03-09 11:48:54,898 : ERROR : Error verifying l10n dashboard changesets

• reran

# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check --dryrun localhost:9001

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check localhost:9001

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

And for 11.0 final it's

Firefox,firefox,firefox,11.0,11.0

E-mail mirrors@mozilla.org

Not necessary for rapid release betas, files only go as far as the 'internal mirrors'.

Coordinate yourself with the person creating the ESR and the latest release which normally go out at the same time.

They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time.

Subject: Firefox 11.0 [and {VERSION2}] coming to mirrors on {DATE}
Body:
We're planning on pushing our Firefox 11.0 [and {VERSION2}] release(s) to mirrors 24 hours before {DATE}.
# If this is a chemspill, please mention that in the email as it will mean pinging 
# for CDN to be enabled when the push to internal mirrors happens

If you believe these releases or these dates will cause any problems,
please notify release at mozilla.com. If you have any problems with mirror
status/weights/etc, please notify mirror-submissions at mozilla.org.

Edit rsync exclude files

Edit /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on stage so that it excludes the current release. This prevents external mirrors from picking up the new release prematurely.

For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.

Tag

• Failed due to slow hg.m.o. Restarted automation from the beginning.

Bouncer Submitter

No problems

Source

No problems

Build

No problems

L10N Repacks

No problems.

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Used this script

Partner Repack

No problems

L10N verify

win32 timed out. rebuilt

Checksums

No problems

Updates

No problems

Update verify

No problems

Reset reserved slaves

echo "0" >master/reserved_slaves_bm07-build1

Check permissions / AV scan

No problems

Push to internal mirrors

Forced the builder. No problems.

Mirrors Policy

Before Release Day

Run backupsnip for release snippets

Do this the night before release day as it can take up to two hours.

Final verification

No problems.

Push index file to mirrors

Added the following lines to the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

+ firefox
+ firefox/releases
+ firefox/releases/11.0
+ firefox/releases/11.0/*/
+ firefox/releases/11.0/*/*/
+ firefox/releases/11.0/*/*/*/
+ firefox/releases/11.0/**index.html
- firefox/releases/11.0/**

Removed the previously added "- firefox/releases/11.0" line. After that, ran the following to lay out the index files:

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/11.0
wget --no-check-certificate -O index.html https://bugzilla.mozilla.org/attachment.cgi?id=540139
sed -i -e 's/5/11/g' index.html 
for dir in `find . -mindepth 1 -type d `; do cp -pv index.html $dir/; done

Remove from internal mirrors

Respin declared for the graphics blocklist crashes not actually being fixed in 11.0b7/11.0 build1 code.

Deleted firefox/releases/11.0 from the mirrors, and filed bug 734692 to speed up clean up of pv-mirror01/02 (the "internal mirrors").

Build 2

Preparing to start Automation

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
• Reserve slaves

echo "10"> master/reserved_slaves_bm08-build1

• Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags (FIREFOX_11_0_RELEASE FIREFOX_11_0_BUILD2)
• update and reconfigure the master
• start automation

# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --dryrun localhost:9001

• Failed because beta-only locales was listed on the l10n dashboard:

2012-03-09 11:48:54,898 : ERROR :       locale mn missing in config
2012-03-09 11:48:54,898 : ERROR :       locale sw missing in config
2012-03-09 11:48:54,898 : ERROR : Error verifying l10n dashboard changesets

• reran

# Dry-run
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check --dryrun localhost:9001

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u rail \
  -V 11.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
  --products firefox --l10n-dashboad-version=11 --bypass-l10n-check localhost:9001

Edit rsync exclude files

Done in build1

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Build

No problems

L10N repacks

No problems

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Used this script

Partner Repack

No problems

L10N verify

No problems

Checksums

No problems

Updates

No problems. create_buildN_snippets step went red what is expected.

Update verify

No problems

Reset reserved slaves

echo "0" >master/reserved_slaves_bm07-build1

Check permissions / AV scan

No problems

Push to internal mirrors

Mirrors Policy

Forced

Before Release Day

Assuming this is not a chemspill.

Run backupsnip for release snippets

~/bin/backupsnip Firefox-11.0-build2
# 68m3.698s

Final verification

No problems.

Push index file to mirrors

N/A

Push to external mirrors

Removed previously added exclusion lines.

Release Day

Push XULRunner to Mirrors

# ssh -i ~/.ssh/xrbld_dsa xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
 --exclude 'jsshell-*' \
 /pub/mozilla.org/xulrunner/nightly/11.0-candidates/build2/ /pub/mozilla.org/xulrunner/releases/11.0/

Update XULRunner wiki page

For major releases or chemspills, update the links on:

• https://developer.mozilla.org/en/XULRunner
• https://developer.mozilla.org/en/Gecko_SDK

Push to Release Channel

Enabled manual-only-updates throttling by this patch

~/bin/pushsnip Firefox-11.0-build2
# 68m46.449s

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
ln -s 11.0 latest-11.0
rm latest-10.0
rm latest
ln -s latest-11.0 latest

Update browser choice website

Filed bug 735348

Problems with Bouncer/sentry

See bug 735363 for the details.

• Forgot to remove "- firefox/11.0/**" line (fallout from build1)
• Forgot to add 11.0 to rsyncd-mozilla-current.exclude

Detached GPG signing for Mac dmg files

• it turned out that we don't upload *dmg.asc files, see bug 738349
• On keymaster:

mkdir -p ~/signing-work/firefox-11.0/build2
cd ~/signing-work/firefox-11.0
rsync -avP --include='*.dmg' --include='*/' --exclude='*' cltbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/ ./build2/mac/
find build2/mac -type f -name '*.dmg' -exec gpg --no-use-agent --batch --passphrase "$(GPGPASS)" -sba '{}' \;
find build2/mac -type f -name '*.dmg.asc' -exec chmod 644 '{}' \;
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/11.0-candidates/build2/mac/
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avP --include='*.dmg.asc' --include='*/' --exclude='*' ./build2/mac/ ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/releases/11.0/mac/