Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

Build Engineers

bhearsum - Tracking bug: bug 731385

Signed-off Revision(s)

Build 1: f66f339fd046

L10N changesets

Left unchanged from 3.6.27

Tags

Build #	Branch, Tags	Changeset
1	GECKO19228_2012030606_RELBRANCH, FIREFOX_3_6_28_BUILD1 FIREFOX_3_6_28_RELEASE	f66f339fd046

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120306064154	mv-moz2-linux-ix-slave05	38 mins, 3 secs
	Mac		bm-xserve16	1 hrs, 40 mins, 54 secs
	Windows		mw32-ix-slave03	1 hrs, 47 mins, 48 secs

Notes

Build 1

Preparing to start Automation

• Set clobbers for "any master", "mozilla-1.9.2", "any builder"
• Reserve slaves
• Landed automation configs
• Tagged buildbot-configs, buildbotcustom, & tools with FIREFOX_3_6_28_{RELEASE,BUILD1} tags
• update and reconfigure the master
• start automation

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.28 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --bypass-l10n-check --dryrun localhost:9001

• Failed with:

Traceback (most recent call last):
  File "../tools/buildbot-helpers/release_sanity.py", line 403, in <module>
    releaseConfig['mozconfigs'],
KeyError: 'mozconfigs'

• Disabled mozconfig checks and tried again:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.28 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --bypass-l10n-check --dryrun --bypass-mozconfig-check localhost:9001

• Started the automation:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.28 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --bypass-l10n-check  --bypass-mozconfig-check localhost:9001

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,3.6.28,3.6
Firefox,firefox,firefox,3.6.29pre,3.6

E-mail mirrors@mozilla.org

Will be sent out with 11.0

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

PRODUCT=firefox
VERSION=3.6.28
BUILD=1
TAG=FIREFOX_3_6_28_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.2.py

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

• Win32 build failed with a weird error in hg_update, autoretried.

No other problems.

Firefox repacks

No problems.

Unittests / Talos

XULRunner

Source

No problems

Build

No problems

Signing

Updated sign_xulrunner.sh and ran it.

Firefox Signing

No problems

L10N verify

No problems

Updates

No problems. Backupsnip took 1h15min, pushsnip took 1h35min.

Update verify

All green, no problems.

Move to /pub/mozilla.org/firefox/candidates

I forgot to precreate and symlink this directory, so I had to do the following to move the release to the proper place. I did that with the following, _after_ partial MARs had been uploaded, but before the updates builder had finished (to make sure that no automation was trying to download these files at the same time):

# ffxbld@stage
cd /pub/mozilla.org/firefox/candidates
rsync -av /pub/mozilla.org/firefox/nightly/3.6.28-candidates/ 3.6.28-candidates/
cd /pub/mozilla.org/firefox/nightly
mv 3.6.28-candidates{,.bak} && ln -s ../candidates/3.6.28-candidates 3.6.28-candidates
# moved instead of deleted just in case i missed something
mv 3.6.28-candidates.bak ~/

Reset reserved slaves

Reset to 0.

Check permissions / AV scan

No virus problems. Permissions check failed with:

find /pub/mozilla.org/firefox/nightly/3.6.28-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*'

Fixed the permissions with:

# ffxbld@stage
find /pub/mozilla.org/firefox/candidates/3.6.28-candidates/build1/ -mindepth 1 -type d ! -perm 755 ! -path '*/contrib*' ! -path '*/partner-repacks*' -exec echo chmod 755 {} \;
chmod 755 /pub/mozilla.org/firefox/candidates/3.6.28-candidates{,/build1}

Filed bug 733726 for investigating this specific issue and bug 733727 after I noticed that other release directories had similar permissions.

Push files to internal mirrors

Added '- firefox/releases/3.6.28' to /pub/mozilla.org/rsyncd-mozilla-releases.exclude on surf and waited 15min for it to get picked up. Then used "force build" to start the push to mirrors builder.

Run backupsnip for beta snippets

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-3.6.28-build1-beta

Push to beta channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-3.6.28-build1-beta

Major Update Generation

Used "force build" to start builder, no problems.

Update Verify

All green.

Final verification

A couple of not fully synced mirrors, nothing needing action though.

Push to external mirrors

Removed "- firefox/releases/3.6.28" from surf:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude. Bumped /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude from 3.6.27 to 3.6.28.

Run backupsnip for release snippets

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-3.6.28-build1

Push to release channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-3.6.28-build1

And after QA finished testing 3.6.x, the MU:

~/bin/pushsnip Firefox-3.6.28-build1-11.0-build2-MU

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/
rm latest-3.6 && ln -s 3.6.28 latest-3.6

Push XULRunner to Mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip /pub/mozilla.org/xulrunner/nightly/3.6.28-candidates/build1/ /pub/mozilla.org/xulrunner/releases/3.6.28/

Update XULRunner wiki page