Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

bug 676534 - Clobberer broken for Major Update builders
~~bug 676013 - L10nVerify is giving a setupBuild exception~~

Build Engineers

bhearsum - bug 675730

Signed-off Revision(s)

Build 1: 48b6fb82a960

L10N changesets

Firefox

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110803131630	mv-moz2-linux-ix-slave04	37 mins, 8 secs
	Mac		bm-xserve11	2 hrs, 2 mins, 58 secs
	Windows		mw32-ix-slave03	1 hrs, 42 mins, 22 secs

Notes

Build 1

Done on buildbot-master08.

Marked "Any master", "mozilla-1.9.2", "Any builder" for a clobberer
Reserved 7 slaves:

echo 7 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

Landed release config changes
Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_20_{BUILD1,RELEASE}
Updated checkouts on bm08, ran checkconfig
Reconfiged
Ran release_sanity.py, which passed:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.20 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001

Started the release:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.20 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

Sent the following mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.6.20,3.6
Firefox,firefox,firefox,3.6.21pre,3.6

Tag

No problems

Start autosign

Once mozilla-1.9.2 was tagged, on keymaster, started signcodepwd.exe, opened a cygwin terminal and ran the following:

PRODUCT=firefox
VERSION=3.6.20
BUILD=1
TAG=FIREFOX_3_6_20_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.2.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Bouncer Submitter

No problems

Source

No problems

Build

Firefox

No problems

Unittests / Talos

All unittests were green. tscroll on Fedora was orange because the browser froze. We don't run that test on regular checkins for 1.9.2, so I decided to ignore it.

XULRunner

Source

No problems

Build

No problems

Signing

Signed with:

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=3.6.20
BUILD=1
TAG=FIREFOX_3_6_20_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Repacks

Firefox

No problems

Firefox Signing

No problems

L10N verify

Didn't fire due to bug 676013. Reconfiged to pick up that bug (it had already been moved into production), then used force build to retrigger. No locale differences, as expected.

Updates

No problems

Update verify

A couple of chunks failed due to bug 664539. Retriggered them and they passed. Didn't inspect the logs further.

3.6.18 -> 3.6.20 Partials

Because most of our Windows and Linux users are on 3.6.18 still, we need to do a manual updates run to generate partial MARs for them, and then merge the snippets together. Done on linux-ix-slave15, because it already had 3.6.20 MARs on it:

# cltbld@aus2-staging
# Make copies of the existing snippet dirs, because we'll be uploading to them
cd /opt/aus2/snippets/staging
rsync -av Firefox-3.6.20-build1-test/ Firefox-3.6.20-build1-test-pre-3.6.18-partials/
rsync -av Firefox-3.6.20-build1-beta/ Firefox-3.6.20-build1-beta-pre-3.6.18-partials/
rsync -av Firefox-3.6.20-build1/ Firefox-3.6.20-build1-pre-3.6.18-partials/

# cltbld@linux-ix-slave15
/builds/slave/rel-m-192-updates/build/patcher-configs
wget --no-check-certificate -Omoz192-branch-from-3.6.18-patcher2.cfg "https://bugzilla.mozilla.org/attachment.cgi?id=550727"
cd ..
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-from-3.6.18-patcher2.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-from-3.6.18-patcher2.cfg
find temp/firefox/3.6.18-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1 -type f -exec chmod 644 '{}' \;
find temp/firefox/3.6.19-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1 -type d -exec chmod 755 '{}' \;
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' temp/firefox/3.6.18-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1/update/ ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.20-candidates/build1/update/
rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1-test/
rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1-beta/
rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1/

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-3.6.20-build1-test
~/bin/pushsnip Firefox-3.6.20-build1-test

As expected, the output from pushsnip only mentioned 3.6.18 snippets being overwritten.

Update Verify

Retriggered chunks 1 & 6 of update verify for each platform (chunk #6 contains full tests for de, en-US and ru; chunk #1 contains quick tests for all other locales). All green. Looked over a couple for sanity, and they appeared to be doing the right thing (downloading & applying 3.6.18-3.6.20 MARs for partials, using complete MARs for completes).

Add Bouncer Entries

Because this is a set of differently named files, we need special Bouncer entries for them. Manually created "Firefox-3.6.20-Partial-3.6.18" and added locations for each OS to it.

Major Update

Used "Force Build" to start the "release-mozilla-1.9.2-major_update" builder.

First attempt failed because the build directory already existed (not sure why clobberer didn't get rid of it....).
- Marked "release-mozilla-1.9.2-major_update" for a clobberer and tried force build again.
Second attempt still didn't clobber on the second time (also had issues because I didn't back out the update verify changes).
- Filed bug 676534 and clobbered mv-moz2-linux-ix-slave05:/builds/slave/rel-m-192-mu manually instead.
- Backed out the build/tools changes, then tried force build again.

No problems on the third attempt.

Update verify

No updates found for as, ka, or oc, because they were dropped between 3.6 and 5.0. No problems with any other locales, didn't look over the logs in further detail.

Socorro Product Data Updates

Added 3.6.20 and 3.6.21pre with the crash stats admin page

Check permissions / AV scan

No viruses or permissions issues.

Triggered a second time after creating 3.6.18 partials

Reset reserved slaves

echo 0 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

Push to beta

Ahead of time, ran backupsnip:

# cltbld@aus2-staging
~/bin/backupsnip Firefox-3.6.20-build1-beta
~/bin/pushsnip Firefox-3.6.20-build1-beta

I should've done this right away, but on August 12 I unthrottled the beta channel with this AUS config update.

Push to internal mirrors

Pushed to internal mirrors on Monday morning (~8am PDT) to hopefully speed up the external push. Added '- firefox/releases/3.6.20' to /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on surf. After waiting 10 minutes, started the push to mirrors builder with the following properties:

release_config: mozilla/release-firefox-mozilla-1.9.2.py
script_repo_revision: FIREFOX_3_6_20_RELEASE

Push XULRunner to Mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/3.6.20-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/3.6.20/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.2.19

...to exclude the previous release. </pre>

Update XULRunner wiki page

Updated XULRunner wiki pages: https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
https://developer.mozilla.org/en/Gecko_SDK

Push to external mirrors

Because the files have already been copied to the releases directory, I removed the following line from /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to start the push:

- firefox/releases/3.6.20

Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to carry 3.6.20 instead of 3.6.19.

Final verification

Win32 was green
Mac was red - there were three '503 Service Temporarily Available' errors from 3347-mozilla.voxcdn.com. Lots of other good requests afterwards, so ignored
Linux was red - another 503, and four 404's from www.mirrorservice.org. Disabled the latter until they finish syncing up all their nodes, and commented on bug 632283

Push to Release Channel

Legneato would like to push the MU, even though it's 3.6.20 --> 5.0.1, so that people still have a path if they go Help > Check for Updates after getting 3.6.20. QA is pending on that path, but throttling is already in place. About a week after 3.6.20 we'll do 3.6.20 --> 6.0.1, but dependent on billboards from bug 668108.

# cltbdl@aus2-staging
~/bin/backupsnip Firefox-3.6.20-build1
~/bin/pushsnip Firefox-3.6.20-build1

Update Symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-3.6 && ln -s 3.6.20 latest-3.6

Push 3.6.20 -> 5.0.1 MUs

We didn't push these on release day, but after the fact did:

# cltbld@aus2-staging
~/bin/backupsnip Firefox-3.6.20-build1-5.0.1-build1-MU
~/bin/pushsnip Firefox-3.6.20-build1-5.0.1-build1-MU
~/bin/pushsnip Firefox-3.6.20-build1-5.0.1-build1-MU-beta

3.6.20 -> 6.0 MU

Generation

This was done well after the original release because release notes were not ready at release time bug 668108.

Landed release config update & reconfiged master
Used "Force Build" to start major update automation
Killed it with "stop build" after noticing that the patcher/update verify config bump steps were missing
- bm08 had a local patch disabling these, probably leftover from 7.0b1
Backed out the local changes, reconfiged
Used "force build" again
Automation completed successfully, but detailsURL properties of snippets didn't have the locale substituted in.
- This is because we the URL "http://www.mozilla.com/%LOCALE%/firefox/latest/details/from-3.6.html" was used - and patcher requires %LOCALE% to be all lowercase.

To fix, did the following:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
rsync -av Firefox-3.6.20-build1-6.0-build1-MU-test/ Firefox-3.6.20-build1-6.0-build1-MU-test.bak/
for snip in `find Firefox-3.6.20-build1-6.0-build1-MU-test -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done
rsync -av Firefox-3.6.20-build1-6.0-build1-MU-beta/ Firefox-3.6.20-build1-6.0-build1-MU-beta.bak/
for snip in `find Firefox-3.6.20-build1-6.0-build1-MU-beta -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done
rsync -av Firefox-3.6.20-build1-6.0-build1-MU/ Firefox-3.6.20-build1-6.0-build1-MU.bak/
for snip in `find Firefox-3.6.20-build1-6.0-build1-MU -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done
# should have no output
grep -R '%LOCALE%' Firefox-3.6.20-build1-6.0-build1-MU* | grep -v '.bak'

# re-push test snippets
 ~/bin/pushsnip Firefox-3.6.20-build1-6.0-build1-MU-test

Update Verify

All green, didn't look at the logs in detail.

Build #	Branch, Tags	Changeset
1	GECKO19220_2011080310_RELBRANCH	48b6fb82a960
1	FIREFOX_3_6_20_BUILD1 FIREFOX_3_6_20_RELEASE	fec2e8213a9d

Releases/Firefox 3.6.20/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Notes

Build 1

E-mail Metrics

Tag

Start autosign

Bouncer Submitter

Source

Build

Firefox

Unittests / Talos

XULRunner

Source

Build

Signing

Repacks

Firefox

Firefox Signing

L10N verify

Updates

Update verify

3.6.18 -> 3.6.20 Partials

Update Verify

Add Bouncer Entries

Major Update

Update verify

Socorro Product Data Updates

Check permissions / AV scan

Reset reserved slaves

Push to beta

Push to internal mirrors

Push XULRunner to Mirrors

Update XULRunner wiki page

Push to external mirrors

Final verification

Push to Release Channel

Update Symlinks

Push 3.6.20 -> 5.0.1 MUs

3.6.20 -> 6.0 MU

Generation

Update Verify

Navigation menu

Search