Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• bug 734493 - release tagging for a build 2 off default doesn't re-bump version
• bug 734494 - release_sanity.py errors out on unexpected versions

Build Engineers

{aki} - Tracking bug: bug 730856

Signed-off Revision(s)

• Build 1: cdc9bf49ac24
• Build 1 2nd go: 039e4d5bb2ce
• Build 2: a3a55f56836d

L10N changesets

None for ESR.

Tags

Build #	Branch, Tags	Changeset
1	MOBILE1003_2012030217_RELBRANCH, FENNEC_10_0_3esr_RELEASE, FENNEC_10_0_3esr_BUILD1	e98557ad5baf
	GECKO1003_2012030217_RELBRANCH, FIREFOX_10_0_3esr_BUILD1 FIREFOX_10_0_3esr_RELEASE	e4050d1b49c9
2	GECKO1003_2012030913_RELBRANCH, FIREFOX_10_0_3esr_BUILD2 FIREFOX_10_0_3esr_RELEASE	26f11b3e7226

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120302182214	mv-moz2-linux-ix-slave11	1 hrs, 45 secs
	Linux64		moz2-linux64-slave08	3 hrs, 59 mins, 47 secs
	Mac		moz2-darwin10-slave06	3 hrs, 12 mins, 37 secs
	Windows		mw32-ix-slave04	3 hrs, 26 mins, 22 secs
2	Linux	20120309135702	mv-moz2-linux-ix-slave14	1 hrs, 12 mins, 13 secs
	Linux64		moz2-linux64-slave01	2 hrs, 50 mins, 20 secs
	Mac		moz2-darwin10-slave44	3 hrs, 24 mins, 21 secs
	Windows		mw32-ix-slave20	3 hrs, 8 mins, 9 secs

Fennec

Build #	Type	Build ID	Build machine	Time to build
1	linux-android	20120302180724	mv-moz2-linux-ix-slave06	1 hrs, 44 mins, 57 secs

Notes

Build 1

Preparing to start Automation

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
• Reserve slaves
• For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# ffxbld@stage
# The following command should have been run 
# mkdir /pub/mozilla.org/firefox/candidates/10.0.3esr-candidates
cd /pub/mozilla.org/firefox/nightly/
ln -s ../candidates/10.0.3esr-candidates .

• Landed automation configs
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags

hg tag -f {FIREFOX,FENNEC}_10_0_3esr_{RELEASE,BUILD1}

• Landed a patch to production-masters.json to add mozilla-esr10 to the mobile_release_branches for bm07; retagged build/tools.
• Lukas told me to wait because there was talk about another fix landing.
• Got review for release sanity patches. Landed one two and the mobile mozconfigs. Re-tagged tools.
• Updated revision. Re-tagged buildbot-configs.

• update and reconfigure the master
• start automation

Dry run with -l to disable l10n dashboard checks:

cd master
. ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.3esr --branch mozilla-esr10 --build-number 1 -c release-firefox-mozilla-esr10.py -c release-fennec-mozilla-esr10.py --dryrun -l --products firefox,fennec localhost:9001

Rerun without --dryrun

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.3esr --branch mozilla-esr10 --build-number 1 -c release-firefox-mozilla-esr10.py -c release-fennec-mozilla-esr10.py -l --products firefox,fennec localhost:9001

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,10.0.3esr,10.0esr
Fennec,firefox-mobile,fennec,10.0.3esr,10.0esr

Tag

No problems.

• firefox mv-moz2-linux-ix-slave09
• fennec mv-moz2-linux-ix-slave06

Bouncer Submitter

No problems.

Source

• make upload failed because the symlink created above was pointing to not existing directory
• fixed

# ffxbld@stage
mkdir /pub/mozilla.org/firefox/candidates/10.0.3esr-candidates

• Forced a rebuild, which failed trying to hg clone to already existing directory
• Clobbered the work directory

# cltbld@mv-moz2-linux-ix-slave21
rm -rf /builds/slave/rel-m-esr10-firefox-source

• Forced another rebuild

Build

Firefox

No problems.

Firefox repacks

No problems.

Fennec

No problems.

Unittests / Talos

Android Signing

Followed MobileSigning:

# cltsign@signing1
cd signing-work/mozharness
hg pull -u
cd ..
python26 mozharness/scripts/sign_android.py --cfg signing/android_mozilla-esr10.py

L10N verify

• some of the builders timed out, forced a rebuild
• win32 verification raised warnings due to low disk space (bug 732516), cleaned up manually and forced a rebuild

Checksums

No problems.

Updates

No problems.

Update verify

No problems.

Reset reserved slaves

Done.

Check permissions / AV scan

No problems.

Going to Beta

Run backupsnip

~/bin/backupsnip Firefox-10.0.3esr-build1
# 22m43.795s

Push to beta

Need build 2.

Before Release Day

Push files to internal mirrors

Forced builder.

Final verification

No problems.

Release Day

Publish Fennec to the Android Market

Push the files

created a push_fennec_esr.sh and ran it.

Android Market

pushed.

Build 2

Preparing to start Automation

• Set clobbers for the any/mozilla-esr10/any

• Reserved 10 slaves

• deleted firefox/releases/10.0.3esr and get IT to rsync without --max-delete (bug 733623.

• Landed a patch to move 10.0.3esr to bm08 to avoid resource/reconfig conflicts with 11.0
• Landed configs
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags

hg tag -f FIREFOX_10_0_3esr_{RELEASE,BUILD2}

• Accidentally tagged FENNEC_10_0_3esr_{RELEASE,BUILD2} as well; fixed via

hg tag --remove FENNEC_10_0_3esr_BUILD2
hg tag -r 7145411cb343 -f FENNEC_10_0_3esr_RELEASE

• updated and reconfigured the master.
  • had to add the following to the Makefile:

(python buildbot-configs/update-master-json.py $(PRODUCTION_MASTERS) master/master_config.json)

• • had to softlink ../buildbot-configs/mozilla/*esr10* into master/ on bm08

• start automation

Dry run with -l to disable l10n dashboard checks:

cd master
. ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.3esr --branch mozilla-esr10 --build-number 2 -c release-firefox-mozilla-esr10.py --dryrun -l --products firefox localhost:9001

Got

2012-03-09 12:26:00,359 : ERROR : config/milestone.txt has incorrect version '# Holds the current milestone.
# Should be in the format of
#
#    x.x.x
#    x.x.x.x
#    x.x.x+
#
# Referenced by milestone.pl.
# Hopefully I'll be able to automate replacement of *all*
# hardcoded milestones in the tree from these two files.
#--------------------------------------------------------

10.0.4esrpre
' (expected '10.0.3')
2012-03-09 12:26:06,829 : ERROR : browser/config/version.txt has incorrect version '10.0.4esrpre
' (expected '10.0.3')
2012-03-09 12:26:13,194 : ERROR : js/src/config/milestone.txt has incorrect version '# Holds the current milestone.
# Should be in the format of
#
#    x.x.x
#    x.x.x.x
#    x.x.x+
#
# Referenced by milestone.pl.
# Hopefully I'll be able to automate replacement of *all*
# hardcoded milestones in the tree from these two files.
#--------------------------------------------------------

10.0.4esrpre
' (expected '10.0.3')

Started automation with -b:

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u aki -V 10.0.3esr --branch mozilla-esr10 --build-number 2 -c release-firefox-mozilla-esr10.py -b --products firefox localhost:9001 2>&1 | tee aki2.out

Tag

Tagging didn't fix the version properly.

Landed that fix and tagged.

Source

mv-moz2-linux-ix-slave20

No problems.

Build

Firefox

No problems.

Firefox repacks

No problems.

Unittests / Talos

L10N verify

No problems.

Checksums

No problems.

Updates

No problems.

Update verify

No problems.

Reset reserved slaves

beta8 started after 10.0.3esr, so left this.

Check permissions / AV scan

Perms were fine.

AV hit a false positive; clam people on IRC fixed this.

Going to Beta

Run backupsnip

time ~/bin/backupsnip Firefox-10.0.3esr-build2
5m33.299s

Before Release Day

Push files to internal mirrors

Forced builder.

Final verification

No problems. I forced this but it looks like it triggered as well.

Release Day

Push to esr

time ~/bin/pushsnip Firefox-10.0.3esr-build2
# real    8m42.921s