Releases/Firefox 3.0.3:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

Tracking release bug

Bonsai queries

Updated CVS Tags devmo page.


Build 1:

Module Branch Tag Pull date
cvsroot/mozilla HEAD FIREFOX_3_0_3_BUILD1 2008-08-27 01:31 PDT
l10n/l10n HEAD FIREFOX_3_0_3_BUILD1 2008-08-27 01:31 PDT

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] fx-win32-1.9-slave2
[Mac compressed] fx-mac-1.9-slave2
[Linux compressed] fx-linux-1.9-slave2


Build 1


Setup before starting:

  • Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
  • On the slaves:
    • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
    • remove /builds/verify/firefox-3.0.1
    • update tinderbox to RELEASE_AUTOMATION_M11
      • For win32:
cd /e/builds/tinderbox/mozilla/tools/tinderbox
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  cp -pv
    • On the linux slave, removed:
      • /builds/updates/firefox-3.0.1 and /builds/updates/3.0.2-test
      • /builds/source/*
      • /data/cltbld/thunderbird-3.0a2*
    • On the mac slave, removed:
      • /builds/tinderbox/Tb-Mozilla1.9-Release/Darwin_8.8.4_Depend
      • /builds/tinderbox/Tb-Mozilla1.9-l10n-Release/Darwin_8.8.4_Depend
      • /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend
  • Space on slaves before starting:
    • fx-linux-1.9-slave2: 11G on /builds
    • fx-mac-1.9-slave2: 24G on /
    • fx-win32-1.9-slave2: 11G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
  • Gulped, and kicked off automation:
buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.3build1 release" go


  • Automated, no problems.


  • Automated, no problems.

Build & Repack

  • mac: Automated, no problems.
  • linux: build went fine, but then "AliveTest" failed because I missed the DISPLAY setting instruction above. This has not been done since the colo power outage a couple of days ago. Fixed missing DISPLAY, restarted build and this time linux build passed.
  • win32: buildbot master crashed out during l10n repack. Manually restarted, patched master config to restart at win32 repack and continue onwards. This worked fine.



Signing failed out on 'de' with:

 Error: Invalid timestamp http address
 Error: TimeStampign Failed. Result = 80070001, (-2147024895)

To work-around, did the following (from

mkdir signed-build6-partial unsigned-build6-partial
cp signed-build1/*en-US* signed-build1-partial/
cp unsigned-build1/*en-US* unsigned-build1-partial/
# copied everything from 'de' -> 'zh-TW' from unsigned-build1 to unsigned-build1-partial
perl ./signing/ -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge the signed builds back together
mv signed-build6-partial/ signed-build6/

...and again when failed on 'fy'.

# then push the builds back to stage (instructions on the signing doc)
Resigning of af->fr (excluding en-US) =

When restarting signing we should've been calling with '-p' to make sure the en-US doesn't get signed again. Unfortunately, the 3.0.2 build notes (where we first had to do this) did not get updated properly. To correct this, we had to resign af->fr so all locales + en-US have the same bits inside. To do so, did the following:

# delete af -> fr from signed-build1
cd ~/signing-work/firefox-3.0.3
cp signed-build1/*en-US* signed-build1-partial/
cp unsigned-build1/*en-US* unsigned-build1-partial/
# copied everything from af -> fr from unsigned-build1 to unsigned-build1-partial/
perl ./signing/ -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} -p 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge back together
mv signed-build1-partial/* signed-build1/
# to verify all of internals are signed the same, did:
mkdir ~/tmp
cd ~/tmp
cp ~/signing-work/firefox-3.0.3/signed-build1/*.exe .
for i in *.exe; do ~/signing-utils/7-zip/7z x $i && FOO=`md5sum nonlocalized/ firefox.exe | awk '{print $1}'` && echo $i: $FOO | tee -a sums.log && rm -rf nonlocalized localized optional removed-files.log setup.exe; done



  • Automated, no problems.

Generate Updates

Need to bring people who already updated to 3.0.2 build5 up to build6, so do this (after the automated run)

# connect as cltbld@fx-linux-1.9-slave2
cd /builds/updates/firefox-3.0.2/patcher/temp/firefox/3.0.1-3.0.2
for dir in aus2.test aus2.beta aus2; do
  # copy complete snippets from 3.0.1 into a new dir for 3.0.2
  mkdir -p b5-b6-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.1/ b5-b6-${dir}/Firefox/3.0.2/
  # use the buildid's from build5
  mv b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008070206  b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008070206         b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008070208         b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008090514
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.test/
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.beta/
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2/

Then go to aus2-staging and push 20080919-Firefox-3.0.2build5-build6-test live.

Publish Updates to Test Channels (betatest & releasetest)

  • Automated, no problems.

Update Verify

  • linux no problems
  • win32 expected problems (known checksum dif bug)
  • mac expected problems (added/dropped locales)


  • Automated, no problems.

Push to beta channel

Backup on aus2-staging doing:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $  time ~/bin/backupsnip 20080925-Firefox-3.0.3-beta
 Running /bin/tar cfvj /opt/aus2/snippets/backup/20080925-3-pre-20080925-Firefox-3.0.3-beta.tar.bz2 .
 real    37m9.378s
 user    0m45.106s
 sys     1m41.298s
 $ time ~/bin/pushsnip 20080925-Firefox-3.0.3-beta
real    2m15.445s
user    0m0.341s
sys     0m19.679s

Sign Installers

Done manually using these installer-signing-instructions here

  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0.3/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0.3/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually added entries to

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0.3/stage-merged/ /home/ftp/pub/firefox/releases/3.0.3/
  • edit the exclude file /pub/ to add the new release.
    • For this release, as FF3.0.2 was so recent, I've left FF3.0.2 in place as well.

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • First testrun hit:
    • 85 occurrences of 404s
    • 11 occurrences of "500 Internal Server" errors. Disabled two mirror sites (, until we contact them and figure this out.
  • Next testrun still had problems with Disabled.
  • Next testrun returned totally clean! :-)

  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd 20080925-Firefox-3.0.3
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080925-Firefox-3.0.3-test/$a");'

Publish Updates to Release Channel

  • In case you need to cancel/abort updates, the quickest way to do this is here

While waiting for formal "go", do the following on aus2-staging:

$ cd /opt/aus2/snippets/staging
$ time ~/bin/backupsnip 20080925-Firefox-3.0.3
real    44m41.791s
user    0m43.892s
sys     1m33.340s
$ time ~/bin/pushsnip 20080925-Firefox-3.0.3

(exact times lost with sudden need to block updates.)

  • QA found problem with url redirect for firstrun page, see bug 457328. This meant that all non en-US builds would get a broken url redirect to the firstrun page.
    • I turned off all updates using instructions above.
    • When redirects were fixed, and I turned back on updates, QA continued to not see updates, until I touched the /opt/aus2/incoming/3 directory. (Caused by bug 430737.) Immediately QA were able to see updates again.
    • I've updated "how to cancel/abort updates" instructions above with this extra step.


  • On stage.m.o, update "latest" symlinks
$ cd /pub/
$ mv latest-3.0 latest-3.0.old
$ ln -s 3.0.3 latest-3.0
$ rm latest-3.0.old