Releases/Firefox 3.0.1:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

Version/config bump bug

Bonsai queries

Updated CVS Tags devmo page.


Build 1:

Module Branch Tag Pull date
cvsroot/mozilla HEAD FIREFOX_3_0_1_BUILD1 2008-07-02 01:52 PDT
l10n/l10n HEAD FIREFOX_3_0_1_BUILD1 2008-07-02 01:52 PDT

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] fx-win32-1.9-slave2
[Mac compressed] fx-mac-1.9-slave2
[Linux compressed] fx-linux-1.9-slave2


Build 1


Setup before starting:

  • Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
  • On the slaves:
  • verify that linux has 'DISPLAY=:0 xhost +' set
  • remove /builds/verify/firefox-3.0rc3 and firefox-3.0rc2
  • update tinderbox to RELEASE_AUTOMATION_M10
    • For win32:
cd /e/builds/tinderbox/mozilla/tools/tinderbox
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  cp -pv
  • On the linux slave, removed:
    • /builds/updates/firefox-3.0rc3
    • /builds/source/*
    • /data/cltbld/firefox-3.0rc3
  • For some reason bm-xserve11 was attached to the buildbot master.
    • Disconnected it, and reconnected the proper slave (bm-xserve10/fx-mac-1.9-slave2
  • Space on slaves before starting:
    • fx-linux-1.9-slave2: 29G on /builds
    • fx-mac-1.9-slave2: 24G on /
    • fx-win32-1.9-slave2: 17G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
  • Disabled everything after Updates::Push in master.cfg (local modification) because additional patcherconfig/update work needs to be done after Updates::execute but before updates::push.
  • Gulped, and kicked off automation:
buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.1build1 release" gogogo


Automated, no problems.


Automated, no problems


  • TinderConfig failed with:
Step TinderConfig died: ASSERT: no replacement found for appVersion at Bootstrap/ line 364, <INFILE> line 287.

Not sure why this happened ('version' should be used when appVersion isn't present, I think. Maybe it's not when oldAppVersion is present?). Added it to the config, checked in, tagged, restarted builds.

  • TinderConfig failed spuriously because en-US configs were already tagged. Commented out TinderConfig from master.cfg and restarted builds.
  • Build failed out on Mac - /usr/bin/python was pointing to python2.3 - repointed it to /tools/python/bin/python (python 2.5). not sure how this happened, or how it worked for 3.0rc1/2/3. maybe related to above comments about bm-xserve11.
  • Build failed out on win32 due to disk space. Deleted some trees on e: and restarted. (10G free @ time of restart)


  • Mac repack was done incorrectly because of the TinderConfig problem mentioned above. Only the en-US tinder-config for mac and win32 was bumped. TinderConfig was manually re-run on win32 and mac. This was done before win32 started repacking (therefore, it did the right thing the first time), but after mac had completed its full repack. All of the mac repacks were deleted from stage, and Repack::Excute and ::Push were re-run. This produced the correct builds.
  • No other problems



No changes

Generate Updates

Mac snippets (3.0rc3 -> 3.0.1)
  • Ran PatcherConfig & Updates::Execute to get Mac snippets.
  • Didn't get a 'aus2.beta' directory (beta + release channel snippets were in the same directory).
  • Fixed that by adding 'beta-dir' to <current-update> stanza in the patcher config
  • Re-ran Updates::Execute - got aus2.beta this time
  • Did a spot check, things looked OK.
Linux & win32 snippets (3.0rc2 -> 3.0.1)
  • Updated patcher config to get linux & win32 partials/snippets (from: 3.0rc2, to: 3.0.1, comment out rc3 release stanza, comment out mac from rc2, comment out 3.0rc2 to 3.0rc3 past-update line)
  • Commented out 'clean update area' so we don't clobber the mac snippets.
  • backup mac snippets:
cp -R 3.0rc3-3.0.1/ 3.0rc3-3.0.1.rc3.mac
cp -R 3.0.rc3-3.0.1/ ~/3.0rc3-3.0.1.rc3.mac
  • Re-ran Updates::Execute to generate linux and win32 partials/snippets
    • Got the following at the bottom of the log:
Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2.test: No such file or directory 
at Bootstrap/Step/ line 110
Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2: No such file or directory 
at Bootstrap/Step/ line 119
    • This is because we're generating snippets for a release that is NOT the 'version' as set in bootstrap.cfg. Manual spotcheck below to make sure everything is ok.
  • Manual spotchecks:
# in aus2.test: look for non-test channels snippets in test channel dir
find . -mindepth 6 -type d | grep -v test # expect no output
# the same thing for beta, and release channel
# in aus2.beta
find . -mindepth 6 -type d | grep -v beta # expect no output
# in aus2
find . -mindepth 6 -type d | grep -v release # expect no output
    • Inspected a few snippets from various places - they look OK.
  • Re-checked-in automatically generated patcher config, kept 'beta-dir beta' line.
Merge snippets

Merged snippets + partials into one directory with the following commands:

mkdir 3.0rc3-3.0.1
rsync -av 3.0rc3-3.0.1.rc3.mac/ 3.0rc3-3.0.1/
rsync -av 3.0rc2-3.0.1/ 3.0rc3-3.0.1/

Restarted automation at Updates::Push

Automation continued normally.

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

  • Linux passed.
  • Mac had expected failure of gu-IN (because we stopped shipping it on Mac at rc1)
  • Win32 had expected failures about binary files


No problems

Push Updates to Beta Channels

-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080703-4-pre-20080703-Firefox-3.0.1-beta.tar.bz2 .

real	28m44.170s
user	0m22.740s
sys	0m33.067s
-bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1-beta

sent 737264 bytes  received 44534 bytes  10354.94 bytes/sec
total size is 588339  speedup is 0.75

real	1m15.405s
user	0m0.107s
sys	0m3.612s

Sign Installers

Done manually using these installer-signing-instructions here

  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0.1/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0.1/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually added entries to
  • Linux/win32 partials were added under the 'Firefox-3.0.1-Partial-3.0rc2' product with filenames like:

(Thanks Nick)

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0.1/stage-merged/ /home/ftp/pub/firefox/releases/3.0.1/

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Two 404s caused by mirrors in a half-synced state
  • One 500 error, seems transient
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20080703-Firefox-3.0.1
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080703-Firefox-3.0.1-test/$a");'

Publish Updates to Release Channel

-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080716-2-pre-20080703-Firefox-3.0.1.tar.bz2 .
real	30m39.986s
user	0m23.497s
sys	0m34.947s
-bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1


# on stage as cltbld
cd /home/ftp/pub/firefox/releases
rm latest
ln -s latest-3.0 latest
rm latest-3.0
ln -s 3.0.1 latest-3.0