Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• bug 727740 - mozconfig comparison assumes in-tree mozconfigs since bug 697611
• bug 727751 - xulrunner source failure on 1.9.2

Build Engineers

nthomas - Tracking bug: bug 727630

Signed-off Revision(s)

Build 1: fd6d19a5ae84

L10N changesets

Left unchanged from 3.6.26

Tags

Build #	Branch, Tags	Changeset
1	GECKO19227_2012021601_RELBRANCH, FIREFOX_3_6_27_BUILD1 FIREFOX_3_6_27_RELEASE	73ffd608342e

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120216013254	mv-moz2-linux-ix-slave05	37 mins, 31 secs
	Mac		bm-xserve16	1 hrs, 46 mins, 55 secs
	Windows		mw32-ix-slave09	1 hrs, 48 mins, 34 secs

Notes

Build 1

Preparing to start Automation

• Set clobbers for "any master", "mozilla-1.9.2", "any builder"
• Reserve slaves
• Update l10n changesets for desktop, mobile No l10n changes for a chemspill
• Landed automation configs
• Tagged buildbot-configs, buildbotcustom, & tools with build & release tags
• update and reconfigure the master
• start automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -p firefox -V 3.6.27 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001

This failed because bug 697611 assumes that there is a mozconfigs dict defined in the release config, filed bug 727740. Worked around by cloning tools to ~cltbld/tools and updating to FIREFOX_3_6_26_RELEASE. Retry:

PYTHONPATH=. python ~/tools/buildbot-helpers/release_sanity.py -u nthomas -p firefox -V 3.6.27 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001
...
2012-02-16 00:48:00,842 : INFO : Comparing l10n changesets on dashboard https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx3.6.27 to on-disk l10n-changesets_mozilla-1.9.2 ...
2012-02-16 00:48:01,977 : ERROR : cannot find l10n dashboard at https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx3.6.27
2012-02-16 00:48:01,977 : ERROR : Error verifying l10n dashboard changesets

The real thing

PYTHONPATH=. python ~/tools/buildbot-helpers/release_sanity.py -u nthomas -p firefox -V 3.6.27 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001 -b

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,3.6.27,3.6
Firefox,firefox,firefox,3.6.28pre,3.6

E-mail mirrors@mozilla.org

Combined message with 10.0.2 sent.

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

PRODUCT=firefox
VERSION=3.6.27
BUILD=1
TAG=FIREFOX_3_6_27_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.2.py

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

No problems

Firefox repacks

No problems

Fennec

N/A

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

Failed

mv mozilla-1.9.2/obj-firefox/dist/source/xulrunner-3.6.27.source.tar.bz2 source/xulrunner-3.6.27.source.tar.bz2
mv: cannot stat `mozilla-1.9.2/obj-firefox/dist/source/xulrunner-3.6.27.source.tar.bz2': No such file or directory

Filed bug 727751.

Manual fix:

# cltbld@moz2-linux-slave36
cd /builds/slave/rel-m-192-xr-source/
mv mozilla-1.9.2/obj-firefox/dist/source/xulrunner-1.9.2.27.source.tar.bz2 source/xulrunner-3.6.27.source.tar.bz2
export POST_UPLOAD_CMD='post_upload.py -p xulrunner -v 3.6.27 -n 1 -c'
export UPLOAD_HOST=stage.mozilla.org
export UPLOAD_SSH_KEY=~/.ssh/xrbld_dsa
export UPLOAD_TO_TEMP=1
export UPLOAD_USER=xrbld
python /builds/slave/rel-m-192-xr-source/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python mozilla-1.9.2/build/upload.py --base-path . source/xulrunner-3.6.27.bundle source/xulrunner-3.6.27.source.tar.bz2

Build

No problems

Signing

Updated sign_xulrunner.sh and ran it.

Partner Repack

N/A

Firefox Signing

No problems

Android Signing

N/A

L10N verify

No problems

Updates

Worked fine, except that backupsnip was going to time out again, so I killed it and did the rest by hand. I skipped the backupsnip to save us some load. Ran the following to finish things up:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-3.6.27-build1-test

# cltbld@mv-moz2-linux-ix-slave02
cd /builds/slave/rel-m-192-updates/build/temp/firefox/3.6.26-3.6.27
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.27-build1-beta
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.27-build1

After the pushsnip completed I forced the update verify builders and check permissions builder. I didn't do the virus scan right away because surf was already starting to get bogged down a bit. See virus scan section for details on it.

Update verify

All green.

Major Update

Used "Force Build" to start the builder.

Major Update verify

All green.

Reset reserved slaves

Reset to 0

Check permissions / AV scan

No permissions problems, no viruses. Virus scan was done on an ix machine, roughly like this:

• Installed clamav and clamav-db
• rsynced files from keymaster01
• Ran the following:

# root
rpm -i clamav*.rpm
freshclam
su - cltbld
hg clone http://hg.mozilla.org/build/tools
python tools/stage/extract_and_run_command.py -j4 clamscan -- 3.6.27/ &> scan.log

Push files to internal mirrors

• Updated exclude file with "- firefox/releases/3.6.27" line.
• Used "Force Build" to start push to mirrors builder.

Hit bug 728472. Copied asc files back in via:

rsync -av --dry-run --ignore-existing --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* /pub/mozilla.org/firefox/nightly/3.6.27-candidates/build1/ /pub/mozilla.org/firefox/releases/3.6.27/
rsync -av --ignore-existing --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* /pub/mozilla.org/firefox/nightly/3.6.27-candidates/build1/ /pub/mozilla.org/firefox/releases/3.6.27/

Final verification

Run backupsnip

~/bin/backupsnip Firefox-3.6.27-build1

Push to external mirrors

• Remove "- firefox/releases/3.6.27" line from mozilla-releases rsync file
• Update rsyncd-mozilla-current.exclude from 3.6.26 to 3.6.27

Push release & beta snippets

~/bin/pushsnip Firefox-3.6.27-build1
~/bin/pushsnip Firefox-3.6.27-build1-beta

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-3.6 && ln -s 3.6.27 latest-3.6

Push XULRunner to Mirrors

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip /pub/mozilla.org/xulrunner/nightly/3.6.27-candidates/build1/ /pub/mozilla.org/xulrunner/releases/3.6.27/

Update XULRunner wiki page

Updated the links on these pages:

• https://developer.mozilla.org/en/Gecko_SDK

Push 3.6.27 -> 10.0.2 release channel MU

cd /opt/aus2/snippets/staging
# no backupsnip per bhearsum
~/bin/pushsnip Firefox-3.6.27-build1-10.0.2-build1-MU