Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

bhearsum/coop
bug 624494

Bugs hit

bug 626164 - Clobber won't work for build1
bug 637886 - throttling patch not ready ahead of time
bug 629256 - we disabled MU for broken locales for build 1, but then didn't do the same for build 2 and build 3. Needed to push out the unmodified MU list due to time constraints, so broken locales will continue to be broken.

Signed-off Revision(s)

Build 1: bc91b067f42f

L10N changesets

Updates l10n-changesets obtained from the l10n shipping dashboard.

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110121161846	linux-ix-slave14	29 mins, 15 secs
	Mac	20110121150727	bm-xserve11	1 hrs, 41 mins, 34 secs
	Windows	20110121150729	mw32-ix-slave02	1 hrs, 18 mins, 25 secs

Notes

Done on pm01

Build 1

Set reserved slaves to 4:

# cltbld@pm01
echo 4 > /builds/buildbot/builder_master1/reserved_slaves_pm01

Clobber won't work for build1 because of bug 626164
- Looked around all of the fast slaves and didn't see anything lying around, so they were clobbered at some point already.
Landed updated automation config
Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_17_RELEASE_FIREFOX_3_5_17_BUILD1
Updated buildbot-configs/buildbotcustom
Ran release_sanity.py w/ the dry run option:

PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010

After it passed, ran it without --dry-run to start the release process:

PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py localhost:9010

Tag

No problems

Source

No problems

Build

Linux failed to upload symbols
- Clobbered (manually) and re-kicked with "Force Build"
Linux failed to upload symbols again
- Catlee pointed out that the host key verification was failing
- Logged onto the slave, manually clobbered, fixed the hostkey with:

# cltbld@linux-ix-slave14
ssh -oBatchMode=no dm-symbolpush01.mozilla.org
# Typed "yes" when prompted

- Started a new build with "Force Build"
No problems with Mac or Windows

L10n Repack

No problems on Mac or Windows
Had to use "Force Build" with the following properties to start the Linux ones, because of the Linux en-US issues:

release_config: mozilla/release-firefox-mozilla-1.9.1.py
script_repo_revision: FIREFOX_3_5_17_RELEASE

XULRunner Source & Build

No problems

Sign

Removed en-US.xpi from candidates directory

Ran the following on the signing machine

PRODUCT=firefox
VERSION=3.5.17
BUILD=1
TAG=FIREFOX_3_5_17_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Signature verification succeeded.

L10n Verify

Differences for all locales/platforms because of the shortened build directories.

Updates

First attempt failed with an ssh key error, on the same slave that had key issues during the Linux build.
- Replaced keys on it with ones from a known good slave:

# cltbld@linux-ix-slave14
su -c 'umount /home/cltbld/.ssh' # multiple times, until fully unmounted
rm -rf .ssh
scp -r cltbld@mv-moz2-linux-ix-slave08:.ssh .

Restarted updates with "Force Build"
- No problems
the updates builder was triggered a third time, but not sure by what/who. It failed early, and should be harmless.

Update Verify

Triggered by hand.

Linux:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\
b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5b99/20090605144815/Linux_x8\
6-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\
5b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Mac:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\
.1/20090715083437/Darwin_Universal-gcc3/de/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5.1/20090715083437/Darwin_Un\
iversal-gcc3/de/betatest/update.xml?force=1
2011-01-24 15:55:38 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/ni\
ghtly/3.5.17-candidates/build1/update/mac/de/firefox-3.5.17.complete.mar [17980\
405/17980405] -> "update/complete.mar" [1]
FAIL: partial updates are larger than complete updates

Win32:

FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\
5.10/20100504093643/WINNT_x86-msvc/pt-BR/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Socorro product data updates

Added '3.5.17' and '3.5.18pre' to crash-stats with this admin page.

Update Bouncer

Added by forcing bouncer_submitter builder

Major Update Generation (-> 3.6.14)

Used 'force build' to start the major_update builder.

Had more problems with linux-ix-slave14. It turns out that this slave was still syncing with staging-puppet, which is why we were having ssh key issues.

I fixed up the slave to sync with production-puppet and the problem went away.

Major Update Verification

mn fails for all platforms, normal and expected.
sl blurts the following, normal and expected:

Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml

Linux:
some path differences:
- ru:

Only in source/firefox/searchplugins: rambler.xml

Mac: the usual mac differences:

diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js
1c1
 < //@line 2 "/builds/moz2_slave/release-mozilla-1.9.1-macosx_build/build/browser/app/profile/channel-prefs.js"
---
 > //@line 2 "/builds/slave/release-mozilla-1.9.2-macosx_build/build/browser/app/profile/channel-prefs.js"
Only in source/Firefox.app/Contents: Plug-Ins
 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib

Windows: the usual .chk file differences:

Only in source/bin: .autoreg
diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js
1c1
< //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.1-win32_build\build\browser\app\profile\channel-prefs.js"
---
> //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.2-win32_build\build\browser\app\profile\channel-prefs.js"
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20110121-Firefox-3.5.17-beta
 real    35m9.724s
 user    0m2.357s
 sys     0m23.525s
time ~/bin/pushsnip 20110121-Firefox-3.5.17-beta
 real    29m36.768s
 user    0m2.182s
 sys     0m34.530s

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.17-candidates/build1/
find . ! -user ffxbld # (anything in contrib/ is ok)
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

contrib/solaris_pkgadd and solaris_tarball were 755 (g+s), which is OK too.

Virus check

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1/
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 2694.758 sec (44 m 54 s)

Major Update Generation (-> 3.6.14 build2)

Used 'force build' to start the major_update builder.

No problems

Major Update Verification

Same "failures" as for build1.

Major Update Generation (-> 3.6.14 build3)

Used 'force build' to start the major_update builder.

No problems

Major Update Verification

Same "failures" as for build1.

Pre-push Checks

Triggered manually from waterfall.

Revision to build: FIREFOX_3_5_17_RELEASE
Property 1 name -> release_config: mozilla/release-firefox-mozilla-1.9.1.py

Push to mirrors

Used the new automatic builder. Failed a few times before realizing I needed to reconfig the master and re-tag the tools repo to pick up the changes (done by Rail)

No problems after that.

Final verify

A few 404s, but nothing serious.

Push updates to release channel

Please use the

20110126-Firefox-3.5.17-bug629256fix directory when pushing to the release channel for 3.5.x -> 3.5.17 updates
20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix directory when pushing the major update from 3.5.17 to 3.6.14

for bug 629256. We drop updates for a few locales which have strings that make the updater crash. NThomas 23:52, 26 January 2011 (PST)

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20110126-Firefox-3.5.17-bug629256fix

time ~/bin/pushsnip 20110126-Firefox-3.5.17-bug629256fix
 real    17m24.216s
 user    0m6.373s
 sys     0m34.334s

Push 3.5.17 -> 3.6.14 MU

Forgot to roll a version update patch for throttling. Got that landed and then pushed in bug 637886.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix
~/bin/pushsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix

But those were the 3.6.14build1 snippets, fix:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110222-Firefox-3.5.17-3.6.14-MU
~/bin/pushsnip 20110222-Firefox-3.5.17-3.6.14-MU

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases/
rm latest-3.5 && ln -s 3.5.17 latest-3.5

Sent the following to metrics@mozilla.com

Firefox,firefox,firefox,3.5.17,3.5
Firefox,firefox,firefox,3.5.18pre,3.5

FIREFOX DONE TO HERE

XULRunner

Builds

Linux: No problems.
Windows: No problems.
Mac: No problems.

Signing

Signed with:

PRODUCT=xulrunner
VERSION=1.9.1.17
BUILD=1
TAG=FIREFOX_3_5_17_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

xrbld@stage

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \
/pub/mozilla.org/xulrunner/nightly/1.9.1.17-candidates/build1/  \
/pub/mozilla.org/xulrunner/releases/1.9.1.17/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.1.16

...to exclude the previous release.

Update XULRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

XR DONE TO HERE

MU to 3.6.17

rail forced the MU last night.

We had to do a fast 3.6.15 release so we had to generate a new MU

# cltbld@aus2
# From any build slave
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta
~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU
~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta
~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU

Build #	Tag	Changeset
1	FIREFOX_3_5_17_RELEASE FIREFOX_3_5_17_BUILD1	bc91b067f42f

Releases/Firefox 3.5.17/BuildNotes

Contents

Notes About Releasing

Build Engineers

Bugs hit

Signed-off Revision(s)

L10N changesets

Tags

Build data

Notes

Build 1

Tag

Source

Build

L10n Repack

XULRunner Source & Build

Sign

L10n Verify

Updates

Update Verify

Socorro product data updates

Update Bouncer

Major Update Generation (-> 3.6.14)

Major Update Verification

Publish updates to beta channel

Final checks before push

Virus check

Major Update Generation (-> 3.6.14 build2)

Major Update Verification

Major Update Generation (-> 3.6.14 build3)

Major Update Verification

Pre-push Checks

Push to mirrors

Final verify

Push updates to release channel

Push 3.5.17 -> 3.6.14 MU

Release

XULRunner

Builds

Signing

Push XULRunner to mirrors

Update XULRunner wiki

MU to 3.6.17

Navigation menu

Search