Releases/Firefox 3.0.4:BuildNotes

Build Engineers

joduinn
Tracking release bug

Bonsai queries

Bonsai query is: last checkin

Updated CVS Tags devmo page.

Tags

Build 1:

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	FIREFOX_3_0_4_BUILD1	2008-10-28 22:06 PDT
l10n/l10n	HEAD	FIREFOX_3_0_4_BUILD1	2008-10-28 22:06 PDT

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				fx-win32-1.9-slave2
[Mac compressed]				fx-mac-1.9-slave2
[Linux compressed]				fx-linux-1.9-slave2

Notes

Build 1

Bootstrap Tag: RELEASE_AUTOMATION_M11

Setup before starting:

• Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
• On all the slaves:
  • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
  • remove /builds/verify/firefox-3.0.2, /builds/verify/firefox-3.0.3
  • verified we're using same version of tinderbox as before (RELEASE_AUTOMATION_M11), so nothing to do.
  • On the linux slave, removed:
    • /builds/updates/firefox-3.0.3
    • /builds/source/firefox-3.0.3
    • /data/cltbld/firefox-3.0.3
  • On the mac slave, removed nothing, as there was loads of space.
• Space on slaves before starting:
  • fx-linux-1.9-slave2: 27G on /builds
  • fx-mac-1.9-slave2: 22G on /
  • fx-win32-1.9-slave2: 16.5G on d:, 2.5G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)

• Gulped, and kicked off automation:

#login to production-1.9-master as buildbot
#
buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.4build1 release" go

Tag

• No problems

Source

• No problems

Build & Repack

• linux, mac: No problems
• win32: slave ran out of space during repack:push.
  • Even though I'd verified that we had enough free space beforehand - and had more free space then we had when starting FF303. Still dont understand why.
  • restarted at just repack:execute, ran to completion just fine.

Sign

• Signing doc
• no problems.
• tweaked master.cfg, and did new sendchange to resume automation from this point.

$ buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.4build1 release after signing" go

L10nVerify

• Automated - no problems.

Generate Updates

• Automated - no problems.

Publish Updates to Test Channels (betatest & releasetest)

• Automated - no problems.

Update Verify

• linux - no problems.
• mac - expected failure on gu-IN for 3.0b1...3.0b5. Otherwise, no problems.
• win32 - first run timed out because the updater tries to elevate its access rights on windows - the OS throws up a modal RunAs dialog when we try to run the updater which blocks any progress. Probably fallout from bug 437349 in Fx3.0.2. This patch fixes it up:

Index: check_updates.sh
===================================================================
RCS file: /cvsroot/mozilla/testing/release/common/check_updates.sh,v
retrieving revision 1.11
diff -u -r1.11 check_updates.sh
--- check_updates.sh    21 Mar 2008 21:06:21 -0000      1.11
+++ check_updates.sh    3 Nov 2008 20:42:16 -0000
@@ -41,7 +41,7 @@
  if [ -d source/$platform_dirname ]; then
    cd source/$platform_dirname;
    cp $updater ../../update
-    ../../update/updater ../../update 0
+    ../../update/updater ../../update 0 .
    cd ../..
  else
    echo "FAIL: no dir in source/$platform_dirname"

This should be fixed by bug 452128 in 3.0.4, so that 3.0.4 -> 3.0.5 updates work without the patch

• a second win32 run failed with the usual complaint that freebl3.chk and softokn3.chk differ (bug 404340), which is a pass in reality

Stage

• Automated - no problems.

Push to beta channel

Done in tandem with 2.0.0.18, see the 2.0.0.18 BuildNotes for details.

Sign Installers

Done manually using these installer-signing-instructions here

• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.4/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

• • FIXME: I noted that the permissions on the *SUMS files were correct before I ran the chmod above. Is this still needed?

Update Bouncer

• Done.
• Note for next release: Do not remove the Check Now bit on the Firefox-3.0.4 Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.4/stage-merged/ /home/ftp/pub/firefox/releases/3.0.4/

• edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.4) and remove the previous release (3.0.3).

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• First testrun hit:
  • ?? occurrences of 404s

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
$ cd 20081103-Firefox-3.0.4
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20081103-Firefox-3.0.4-test/$a");'
$

Publish Updates to Release Channel

While waiting for the formal "go," ran:

time ~/bin/backupsnip 20081103-Firefox-3.0.4
real	33m58.819s
user	0m26.811s
sys	0m37.705s

Release

• In case you need to cancel/abort updates, the quickest way to do this is here

$ time ~/bin/pushsnip 20081103-Firefox-3.0.4

https://bugzilla.mozilla.org/show_bug.cgi?id=466641 was filed on November 25 - latest-3.0 symlink was not updated. I went ahead and fixed this:
# cltbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.4 latest-3.0