Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

whitelist bugs:
Missing l10n dashboard entry
bug 728472 - missing gpg signatures

Build Engineers

nthomas - Tracking bug: bug 727629

Signed-off Revision(s)

Build 1: bd611a3115b0

L10N changesets

None for the chemspill.

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120215223356	mv-moz2-linux-ix-slave11	1 hrs, 1 mins, 54 secs
	Linux64		moz2-linux64-slave08	4 hrs, 5 mins, 30 secs
	Mac		moz2-darwin10-slave45	3 hrs, 4 mins, 9 secs
	Windows		mw32-ix-slave02	3 hrs, 1 mins, 55 secs

Fennec

Build #	Type	Build ID	Build machine	Time to build
1	android-xul	20120215223137	mv-moz2-linux-ix-slave06	1 hrs, 51 mins, 15 secs
	linux		mv-moz2-linux-ix-slave07	1 hrs, 2 mins, 18 secs
	macosx		bm-xserve15	2 hrs, 10 mins, 56 secs
	win32		mw32-ix-slave03	1 hrs, 30 mins, 25 secs

Notes

Build 1

Preparing to start Automation

Set clobbers for "any master", "mozilla-release", "any builder"
set reserved_slaves to 8
Landed configs
Tag buildbot-configs, buildbotcustom, & tools with build & release tags
update and reconfigure the master
start automation

Dry run:

. ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.2 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py -c release-fennec-mozilla-release.py --dryrun --products firefox,fennec localhost:9001 2>&1 | tee aki.out

Saw a bunch of mozconfig errors + l10n dashboard errors, sadly all known+expected.

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.2 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py -c release-fennec-mozilla-release.py -b --products firefox,fennec localhost:9001 2>&1

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,10.0.2,10.0
Fennec,firefox-mobile,fennec,10.0.2,10.0

E-mail mirrors@mozilla.org

Combined message with 3.6.27 sent.

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

PRODUCT=firefox
VERSION=10.0.2
BUILD=1
TAG=FIREFOX_10_0_2_RELEASE
REPO='releases/mozilla-release'
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=mozilla/release-firefox-mozilla-release.py

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

No problems

Firefox repacks

One of the win32 l10n chunks failed to clone pt-PT, ru, and sk, with "Request URI Too Large" errors.
- This happens when there's too many heads in the repository, and the client Mercurial program is old
- Tried to fix by having the l10n scripts clone over ssh, but our helper methods don't support that so gave up
- Eventually fixed by pre-cloning the repositories over ssh by hand, for each locale:

# cltbld@mw32-ix-slave03
d:
cd mozilla-build/msys/bin
bash
export PATH=$PATH:/d/mozilla-build/msys/bin:/d/mozilla-build/hg
cd /e/builds/hg-shared/releases/l10n/mozilla-release
hg clone -e 'ssh -i "/c/Documents and Settings/cltbld/.ssh/ffxbld_dsa"' ssh://hg.mozilla.org/releases/l10n/mozilla-release/pt-PT
# edit .hg/hgrc by hand to switch the path to http://

- Then triggered standalone builder first for just 'ru', to test, and then for 'pt-PT:sk' to finish things off
- After that builder finished had to trigger "repack complete" by hand, which caused the partner repack builder to run, and then signing.

Fennec

No problems

Fennec repacks

Unittests / Talos

Look to see that they ran
Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Updated sign_xulrunner.sh and ran it.

Partner Repack

Firefox

No problems

Fennec

N/A

Firefox Signing

No problems

Android Signing

No problems, updated and ran sign_android-0.8.sh

Android Signature Verification

No problems

L10N verify

No problems

Updates

Backupsnip got timed out, forced it through by hand:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-10.0.2-build1-test

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/temp/firefox/10.0.1-10.0.2
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-10.0.2-build1

Update verify

Swapped load to ftp.m.o from stage.m.o (change), and forced update verify:

for p in linux linux64 macosx64 win32; do
    for i in 1 2 3; do
      echo $p $i
      curl -s "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F3/force" >/dev/null
      sleep 5
    done
done

No problems

Reset reserved slaves

Reset to 0

Check permissions / AV scan

No permissions problems. Did virus scan by hand, similar to https://wiki.mozilla.org/Releases/Firefox_3.6.27/BuildNotes#Check_permissions_.2F_AV_scan

Run backupsnip for release snippets

~/bin/backupsnip Firefox-10.0.2-build1

Push files to internal mirrors

Somone had already added an line to rsyncd-mozilla-releases.exclude to stop 10.0.2 getting to the externals. Forced this builder with script_repo_revision and release_config set as properties, no problems. 1 hour to copy.

Hit bug 728472. Copied asc files back in via:

rsync -av --dry-run --ignore-existing --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* /pub/mozilla.org/firefox/nightly/10.0.2-candidates/build1/ /pub/mozilla.org/firefox/releases/10.0.2/

rsync -av --ignore-existing --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* /pub/mozilla.org/firefox/nightly/10.0.2-candidates/build1/ /pub/mozilla.org/firefox/releases/10.0.2/

Final verification

Had to force the release-mozilla-release-start_uptake_monitoring builder, presumably because updates failed earlier. No problems.

Push to external mirrors

We enabled CDNetworks at 45k to carry the initial load
removed the 10.0.2 line from stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude
updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to have 10.0.2 instead of 10.0
CDN was turned down as mirrors picked up, then off

Release Day

Publish Fennec to the Android Market

Push the files

Used this script and ran as ffxbld@stage

Android Market

IMPORTANT - Make sure to follow instructions!

downloaded multi apk: http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/10.0.2-candidates/build1/android/multi/fennec-10.0.2.multi.android-arm.apk
visit https://market.android.com/publish
choose "Firefox"
select the "APK Files" tab and choose "Upload APK"
- choose the multi apk that you downloaded and hit "upload"
  - NOTE: the progress bar does not seem to show progress (at least on Google Chrome)
- when it finishes uploading, verify VersionCode is the build date (almost anyway, the market reports an hour later than our buildID)
- hit "Save"
activate new apk (You should see an Error message since both apk's are active)
deactivate old apk (You should not see the Error message anymore)
hit "Save"
If this is a beta 1, go to "Product Details" tab -> Recent Changes, change the url to http://www.mozilla.com/en-US/mobile/{VERSION}beta/releasenotes/
- and hit "Save"

Push XULRunner to Mirrors

rsync -avn --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
--exclude 'jsshell-*' /pub/mozilla.org/xulrunner/nightly/10.0.2-candidates/build1/ \
 /pub/mozilla.org/xulrunner/releases/10.0.2/

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
--exclude 'jsshell-*' /pub/mozilla.org/xulrunner/nightly/10.0.2-candidates/build1/ \
/pub/mozilla.org/xulrunner/releases/10.0.2/

Update XULRunner wiki page

Push to Release Channel

~/bin/pushsnip Firefox-10.0.2-build1

and unthrottled.

Update symlinks

For major releases or chemspills, update the links on:

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-10.0 && ln -s 10.0.2 latest

Update browser choice website

Filed bug 728116.

Build #	Branch, Tags	Changeset
1	MOBILE1002_2012021522_RELBRANCH, FENNEC_10_0_2_RELEASE, FENNEC_10_0_2_BUILD1	c386bba0f925
1	GECKO1002_2012021522_RELBRANCH, FIREFOX_10_0_2_BUILD1 FIREFOX_10_0_2_RELEASE	72ad46d416ce

Releases/Firefox 10.0.2/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Fennec

Notes

Build 1

Preparing to start Automation

E-mail Metrics

E-mail mirrors@mozilla.org

Tag

Bouncer Submitter

Source

Start autosign

Build

Firefox

Firefox repacks

Fennec

Fennec repacks

Unittests / Talos

XULRunner

Source

Build

Signing

Partner Repack

Firefox

Fennec

Firefox Signing

Android Signing

Android Signature Verification

L10N verify

Updates

Update verify

Reset reserved slaves

Check permissions / AV scan

Run backupsnip for release snippets

Push files to internal mirrors

Final verification

Push to external mirrors

Release Day

Publish Fennec to the Android Market

Push the files

Android Market

Push XULRunner to Mirrors

Update XULRunner wiki page

Push to Release Channel

Update symlinks

Update browser choice website

Navigation menu

Search