Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

The following are the list of bugs that we expect to hit or that we did not expect to hit. For the next release some of this will be already fixed.

• We forgot that beta4 was not shipped and update verify failed

These two required the manual work for major updates:

• bug 662803 - Major update automation should support multiple "from" versions
• bug 662298 - major update automation should be able to be run independently, or as part of the "to" release

Also check the "known issues" page.

For postmortem

The whole release was rushed up (go to build later on EDT) and expectations of releasing on next afternoon was not explicit. If it was known I could have started things earlier.

Build Engineers

armenzg - bug 662509

Signed-off Revision(s)

Build 1: 3524ac4292a7

L10N changesets

From Axel's webapp.

Tags

Build #	Tag	Changeset
1	GECKO50_2011060814_RELBRANCH	cf9c20531b17
	FIREFOX_5_0b5_RELEASE FIREFOX_5_0b5_BUILD1	cf9c20531b17

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110608151458	mv-moz2-linux-ix-slave13	59 mins, 35 secs
	Linux64		moz2-linux64-slave02	1 hrs, 7 mins, 45 secs
	Mac		moz2-darwin10-slave48	4 hrs, 9 mins, 3 secs
	Windows		mw32-ix-slave17	2 hrs, 42 mins, 39 secs

Notes

Build 1

Setup

• Marked "Any master", "mozilla-beta", "Any builder" for a clobber
• About an hour ahead of the expected "go", set reserved_slaves to 6:

[cltbld@production-master03 builder_master]$ pwd
/builds/buildbot/builder_master
[cltbld@production-master03 builder_master]$ echo 6 > reserved_slaves_pm03 

• Hit "ship it" button on "Firefox 5 Beta Build 5"</i> (https://l10n-stage-sj.mozilla.org/shipping/milestones)
• Landed configs, merged configs & custom to production.
• Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_5_0b5_RELEASE and FIREFOX_5_0b5_BUILD1
• Update pm03 checkouts & reconfigured
• Ran release_sanity dry run with:

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py --dryrun localhost:9010

• And then to start the release:

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u armenzg -V 5.0b5 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py localhost:9010

Tag

Worked fine.

Start Autosign

Once mozilla-beta was tagged, on keymaster, started signcodepwd.exe, opened a cygwin terminal and run the following:

PRODUCT=firefox
VERSION=5.0b5
BUILD=1
TAG=FIREFOX_5_0b5_RELEASE
REPO='releases/mozilla-beta'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-beta.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Bouncer Submitter

Worked fine.

Source

Worked fine.

Build/Repack

• Linux: No problems
• Linux64: No problems
• Windows: No problems
• Mac: No problems

Reset reserved_slaves

[cltbld@production-master03 builder_master]$ pwd
/builds/buildbot/builder_master
[cltbld@production-master03 builder_master]$ echo 0 > reserved_slaves_pm03

Signing

No problems. Closed signcodepwd.exe after signing finished. Got "signature verification succeeded" mail a few hours later.

L10n Verify

A few legitimate looking string changes.

Updates (5.0b[1-4] -> 5.0b5)

No problems.

Update verify

Beta4 had not been shipped and therefore update verify failed.

Process stderr:
http://stage-old.mozilla.org/pub/mozilla.org//firefox/releases/5.0b4/linux-i686/af/firefox-5.0b4.tar.bz2:
22:10:02 ERROR 404: Not Found.

• We landed a switch from releases/5.0b4 to nightly/5.0b4-candidates/build1.
• Tagged with FIREFOX_5_0b5_{RELEASE,BUILD1}_RUNTIME.
• Re-build each one of the failing jobs

It all looks green now.

Updates (4.0many -> 5.0b5)

Manually bumped patcher config and update verify config.

Bumped FIREFOX_4_0_1_RELEASE tag on buildbot-configs, FIREFOX_4_0_1_{BUILD1,RELEASE}_RUNTIME on tools.

Generation of updates was done on mv-moz2-linux-ix-slave13, using the UPDATE_PACKAGING_R14 tag, and re-using the updates directory from the 5.0b[12] -> 5.0b5 updates run, to save time & energy.

Disabled the slave from slavealloc and gracefully shut it down.

cd /builds/slave/rel-beta-updates/build/patcher-configs
cvs up
cd ..
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

cd temp/firefox
ln -s 5.0b5 4.0.1-5.0b5
cd ../..

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

# remove partials for 4.0.1
cd temp/firefox/4.0.1-5.0b5
find aus2{,.test} -name partial.txt -exec rm {} \;

# check releasetest == beta
bash /builds/slave/rel-beta-updates//tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest

# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b5-build1-4.0.1-build1

# push test snippets live
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1-test'

Update Verify

To force all of the verify builders, used the following script (warning: not OS X compatible):

for p in linux linux64 macosx64 win32; do
  for i in `seq 1 10`; do
    curl "http://production-master1.build.sjc1.mozilla.com:8010/builders/release-mozilla-2.0-${p}_major_update_verify_${i}%2F10/force" >/dev/null
  done
done

Differences in all platforms:

 diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
 < //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js"
 < pref("app.update.channel", "release");
 ---
 > //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js"
 > pref("app.update.channel", "beta");
 WARN: non-binary files found in diff

Reviewing this GREEN logs are so verbose. How could this be improved?

Channel switching snippets

We'll use the 4.0.1 snippets for this, which has all of the locales we care about, plus a (harmless) few extra. Like 5.0b2, we put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones.

NOTE: For this to work you have to trigger the MU first.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-5.0b5-build1-channel-switching/Firefox/
rsync -a Firefox-5.0b5-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b5-build1-channel-switching/Firefox/
cd Firefox-5.0b5-build1-channel-switching/Firefox/
mv 4.0.1 5.0
for d in 5.0/*; do 
  mv -v $d/* $d/30000000000000
done

Permissions check and A/V Scan

No problems.

A/V Notification

No problems.

Push Files

As in 5.0b2, we'll be pushing to a few internal mirrors rather than the whole network. To set this up, added the following to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

- firefox/releases/5.0b5

...and waited a few minutes, to ensure that the rsync servers have the file.

Then, forced the "release-mozilla-beta-push_to_mirrors" builder with the following properties:

release_config: mozilla/release-firefox-mozilla-beta.py
script_repo_revision: FIREFOX_5_0b5_RELEASE

Final verify

Triggered "release-mozilla-beta-$PLATFORM_final_verification" without any problems.

Using internal mirrors was so fast!!

Push snippets

"Firefox-5.0b5-build1" and "Firefox-5.0b5-build1-channel-switching" both only have a "5.0" version in them, so we only need one backupsnip between them. "Firefox-5.0b5-build1-4.0.1-build1" has 4.0 versions, and needs its own. As cltbld@aus2-staging, did the following for backups:

cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-5.0b5-build1
~/bin/backupsnip Firefox-5.0b5-build1-4.0.1-build1

And for the pushes:

~/bin/pushsnip Firefox-5.0b5-build1
~/bin/pushsnip Firefox-5.0b5-build1-channel-switching
~/bin/pushsnip Firefox-5.0b5-build1-4.0.1-build1