Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20101026200835	linux-ix-slave13	1hr
	Mac	20101026200743	bm-xserve09	2h5min
	Windows	20101026210630	mw32-ix-slave05	1h53min

Build 1

Done on pm:8011

Set a clobber for pm:8011, any release, all builders with clobberer
Moved to this master
- mv-moz2-linux-ix-slave03
- bm-xserve07, bm-xserve09
- mw32-ix-slave16
landed automation bumps
tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_12_BUILD1 and FIREFOX_3_6_12_RELEASE
updated /tools/buildbotcustom/buildbotcustom/ (buidlbot-0.7 branch) and /builds/buildbot/configs/
removed *.pyc in /builds/buildbot/moz2-master2
reconfiged master
kicked off automation with:

buildbot sendchange --username=nthomas --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.12 build1" Go

Tag

No problems. Landed version bumps on default.

During tagging, catlee also moved over

linux-ix-slave13, mv-moz2-linux-ix-slave08
bm-xserve16
mw32-ix-slave03, mw32-ix-slave05

Source

No problems with Firefox or XULRunner source steps.

Build/Repack

Linux en-US: no problems
Linux repacks: no problems
Mac en-US: no problems
Mac repacks: no problems
Win32 en-US: no problems
Win32 repacks: no problems

XULRunner builds

All green.

Return slaves

Keeping enough for update generation and verification, plus l10n verify.

Partner Repacks

Built, unused.

Unittest results

Unittests never triggered, builders are missing bug 607592

Sign

Removed en-US.xpi from candidates directory

To speed up signing, did the following on keymaster prior to repacks finishing:

PRODUCT=firefox
VERSION=3.6.12
BUILD=1
TAG=FIREFOX_3_6_12_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make df verify-gpg-passphrase setup verify-signcode-passphrase PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
# download target run multiple times to get things as they complete
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

Once all the repacks finished, ran the following in the same window to do the signing:

# Need to download one more time to make sure we've got everything
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
make stubs sign2 stage verify fake-upload PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

L10N Verify

No problems

Updates

3.6* -> 3.6.12

Generation

bug 607564 upload_snippets failed with:

rsync -av -e ssh -oIdentityFile=~/.ssh/cltbld_dsa aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101026-Firefox-3.6.12-test
...
Permission denied (publickey).

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(463) [sender=2.6.8]

Second run failed during commit_verify_configs (bug 607570):

hg commit ...
nothing changed
program finished with exit code 1

Disabled self.bumpVerifyConfigs() in ReleaseUpdatesFactory (http://diff.pastebin.mozilla.org/827934), reconfigured the master, reran updates.

Reverted the patch and reconfigured the master after updates builder finished.

Update Verify

Due to failed updates had to force update verify for all platforms.

Linux: PASS
Mac: PASS
Win32: PASS

3.0.19 -> 3.6.12 MU

Generation

On mv-moz2-linux-ix-slave03

mkdir -p /builds/slave/3.0.19-3.6.12-major-update/snippets
cd !$
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log
# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.12 3.0.19-3.6.12
cd ../..
# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verification

cd /builds/slave/3.0.19-3.6.12-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.12/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

cd /builds/slave/3.0.19-3.6.12-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.12
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101027-Firefox-3.0.19-3.6.12-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101027-Firefox-3.0.19-3.6.12-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101027-Firefox-3.0.19-3.6.12-MU

Enable test snippets

# cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101027-Firefox-3.0.19-3.6.12-MU-test
~/bin/pushsnip 20101027-Firefox-3.0.19-3.6.12-MU-test

Verify

mv-moz2-linux-ix-slave03, mw32-ix-slave03, bm-xserve07

mkdir -p /builds/slave/verify/firefox-3019-3612-major/
cd /builds/slave/verify/firefox-3019-3612-major/
hg clone http://hg.mozilla.org/build/tools

cd tools/release/updates
platform=linux   # or mac
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Results

PASS, identical to those found here: https://wiki.mozilla.org/Releases/Firefox_3.0.19/BuildNotes#Results the 3.0.19->3.6

Socorro product data updates

Added '3.6.12' and '3.6.13pre' to crash-stats with this admin page.

Update Bouncer

Added by forcing bouncer_submitter builder

EU Ballot Build

Given unsigned builds by kev, signing notes in the signing bug.

Staged with:

# ffxbld@stage
# get tarball of unsigned EUballot builds from kev
# RelEng to sign the EUballot builds
# put copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv ../euballot-3612-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/mozilla.org/firefox/nightly/3.6.12-candidates/build1/

Virus Scan

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.12-candidates/build1/

Final Checks Before Push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.12-candidates/build1/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

contrib directories/files not owned by ffxbld, that's OK
partner builds not 644/755, OK because we aren't shipping them. FIXME anyways though
contrib/solaris_pkgadd and solaris_tarball were 755 (g+s), which is OK

Push to mirrors

justdave ran this command directly on ftp01 rather than us running it on stage, because it can go faster when run there:

rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.6.12-candidates/build1/ /pub/mozilla.org/firefox/releases/3.6.12/

Needed to also exclude *.tests.zip too, so came back and removed

linux-i686/en-US/firefox-3.6.12.tests.zip
mac/en-US/Firefox 3.6.12.tests.zip

The signing process excludes the windows copy of that file.

We did this part still:

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.11 with 3.6.12

Final verification (sanity check releasetest snippets)

pj-mirror01.mozilla.org was slow to sync, finished before we were ready to release though, so this is a PASS

Move more slaves back

Last ones:

mv-moz2-linux-ix-slave03
mw32-ix-slave03,16
bm-xserve07

Push release/beta channel snippets

# cltbld@aus2-staging
~/bin/backupsnip 20101027-Firefox-3.6.12
~/bin/pushsnip 20101027-Firefox-3.6.12
~/bin/pushsnip 20101027-Firefox-3.6.12-beta

Push throttling changes

3.0.19 was already throttled, but there are 3.5.x throttling changes in the 3.5.12 notes.

Release

Update symlinks

#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.12 latest-3.6

Sent the following mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.6.12,3.6
Firefox,firefox,firefox,3.6.13pre,3.6

Push EU Ballot build

Manually added Bouncer entry. Tomcat filed the bug to get the websites updated. bug 607792

Some caching issues, bug 607868.

Push 3.0.19 -> 3.6.12 MU snippets

Per release drivers, we decided to ship this unthrottled from the start.

# cltbld@aus2-staging
~/bin/backupsnip 20101027-Firefox-3.0.19-3.6.12-MU
~/bin/pushsnip 20101027-Firefox-3.0.19-3.6.12-MU
~/bin/pushsnip 20101027-Firefox-3.0.19-3.6.12-MU-beta

Filed bug 609364 to hold the AUS config. Filed bug 609365 to get it unthrottled

XULRunner Signing

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.2.12
BUILD=1
TAG=FIREFOX_3_6_12_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.12-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.12/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.2.10

(1.9.2.10 is the most recently released XULRunner, we never pushed 1.9.2.11)

Update XULRunner entries on wiki

Updated the links in these two wiki pages for the version change:

Build #	Tag	Changeset
1	FIREFOX_3_6_12_RELEASE FIREFOX_3_6_12_BUILD1	8fe44c79dfd1

Releases/Firefox 3.6.12/BuildNotes

Contents

Notes About Releasing

Build Engineers

Signed-off Revision(s)

Tags

Build data

Build 1

Tag

Source

Build/Repack

XULRunner builds

Return slaves

Partner Repacks

Unittest results

Sign

L10N Verify

Updates

3.6* -> 3.6.12

Generation

Update Verify

3.0.19 -> 3.6.12 MU

Generation

Quick Verification

Push to AUS

Enable test snippets

Verify

Results

Socorro product data updates

Update Bouncer

EU Ballot Build

Virus Scan

Final Checks Before Push

Push to mirrors

Final verification (sanity check releasetest snippets)

Move more slaves back

Push release/beta channel snippets

Push throttling changes

Release

Push EU Ballot build

Push 3.0.19 -> 3.6.12 MU snippets

XULRunner Signing

Push XULRunner to mirrors

Update XULRunner entries on wiki

Navigation menu

Search