Releases/Firefox 3.0.8/BuildNotes

Build Engineers

bhearsum,nthomas
Tracking release bug

Bonsai queries

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	FIREFOX_3_0_8_BUILD1
l10n/l10n	HEAD	FIREFOX_3_0_8_BUILD1

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				fx-win32-1.9-slave2
[Mac compressed]				fx-mac-1.9-slave2
[Linux compressed]				fx-linux-1.9-slave2

Notes

Build 1

Setup before starting:

• On linux slave (fx-linux-1.9-slave2):
  • Deleted /builds/verify/firefox-3.0.7
  • Deleted /builds/updates/*
  • Deleted /builds/source/*
  • Deleted /builds/data/cltbld/firefox-3.0.7
  • Deleted /builds/tags/*

• On mac slave (fx-mac-1.9-slave2):
  • Deleted /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend
  • Deleted /builds/verify/firefox-3.0.7

• On windows slave (fx-win32-1.9-slave2):
  • Deleted /e/xr19rel/WINNT_5.2_Depend

• Space on slaves before starting:
  • fx-mac-1.9-slave2: 19GB on /
  • fx-linux-1.9-slave2: 19GB on /
  • fx-win32-1.9-slave2: 1.5GB on E, 16GB on D (E gets cleaned up at the start of the en-US and l10n builds, so this should be OK).

• Kicked off automation:

buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.8build1 release" go

Tag

Died out because the relbranch wasn't bumped to 3.0.8pre. Checked in version bumps to milestone.txt, version.txt, and module.ver, relbranch, deleted /builds/tags/FIREFOX_3_0_8_RELEASE, restarted.
Had to stop it because I forgot to bump the second version number in module.ver. Had to untag some files and commit s/3.0.8/3.0.8pre/ in the version files before starting again.
Died again because client.mk wasn't commited. Had to do roll back the relbranch version to 3.0.8pre, and modify MOZ_CO_TAG in client.mk to make it work.

Did rdiff from FIREFOX_3_0_7_RELEASE to FIREFOX_3_0_8_RELEASE and got the expected version bumps (4 files), code changes (2 files). There was also a bunch of added tests, Camino changes, and missing mozilla/themes. Build logs indicate that the lattermost is not used.

Source

Automated, no problems.

Build & Repack

Automated, no problems.

Respin declared

Build 2

• Deleted /builds/source/*, /builds/tags/FIREFOX_3_0_8_BUILD1 on fx-linux-1.9-slave2

• Kicked off automation:

buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.8build2 release" go

Tag

No problems

Source

No problems

Build & Repack

No problems

Sign

• Signing doc
• Signed en-US first to help speed up QA process
• Signed locales after they were done, using {un,}signed-build2-resign/ dirs. Pushed up both signing logs, checked en-US wasn't uploaded etc.

L10nVerify

No problems.

Generate Updates

No problems

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

Linux failed because pushing the verify config back up to hg.m.o would have created a new head - this is race against the other platforms. Removed fx-linux-1.9-slave2:/builds/verify/firefox-3.0.8/* and forced a new run.

Linux and mac were green. Windows red with the normal binary file differences (partial: freebl3.chk, softokn3.chk; complete: partial + added .autoreg and removed-files). Passed

Stage

Extra signing log caused a problem

Step Stage died: ASSERT: IsValidLocaleDeliverable(): Unknown file type in tree: /data/cltbld/firefox-3.0.8/batch1/stage-unsigned/win32_signing_build2-resign.log at Bootstrap/Step/Stage.pm line 660.

Appended that to win32_signing_build2.log and removed it. Removed fx-linux-1.9-slave2:/data/cltbld/firefox-3.0.8. Also only 9.7GB free before restarting, so removed some old tagging dirs from Firefox 3.0 & 3.0.2, and the xulrunner build of 3.0.7. Then reran.

Sign Installers

Done manually using these installer-signing-instructions here.

• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.8/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.8/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Done.
• Note for next release: Do not remove the Check Now bit on the Firefox-3.0.7 Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.8/stage-merged/ /home/ftp/pub/firefox/releases/3.0.8/

• edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.8) and remove the previous release (3.0.7).

Push updates to beta channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20090327-Firefox-3.0.8
real	40m34.586s
user	0m37.708s
sys	0m46.487s
time ~/bin/pushsnip 20090327-Firefox-3.0.8-beta
real	1m19.170s
user	0m0.196s
sys	0m6.042s

Final Verification

• Verify that releasetest points to valid bounce links:

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• • A few errors due to partially synced mirrors. By the time I went around to look at them they'd all recovered.

• Before pushing final updates, verify that "release" and "releasetest" channel match:

# on aus2-staging
$ cd /opt/aus2/snippets/staging/20090327-Firefox-3.0.8
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090327-Firefox-3.0.8-test/$a");'
$

Publish Updates to Release Channel

Didn't run backupsnip because we pushed release and beta snippets only a few hours apart.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip 20090327-Firefox-3.0.8

Release

# cltbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.8 latest-3.0