Releases/Firefox 3.1b1:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

release tracking bug

Signed-off Revision


Build 1:

Module Tag Changeset
mozilla-central FIREFOX_3_1b1_BUILD1 FIREFOX_3_1b2_RELEASE

All l10n repositories listed in l10n-changesets were tagged with FIREFOX_3_1b1_RELEASE:

Build data

Type Build ID Build machine
[Windows installer/zip] 20081007144708 moz2-win32-slave11
[Mac compressed] 20081007125523 bm-xserve19
[Linux compressed] 20081007112159 moz2-linux-slave05


  • Cleared up disk space before starting.
    • Removed old actionmonkey directories on linux slaves
  • Free space before starting:
    • moz2-linux-slave01: 13G free
    • moz2-linux-slave02: 16G free
    • moz2-linux-slave03: 16G free
    • moz2-linux-slave05: 12G free
    • moz2-linux-slave06: 16G free
    • bm-xserve16: 25Gi free
    • bm-xserve17: 28Gi free
    • bm-xserve18: 25Gi free
    • bm-xserve19: 23Gi free
    • moz2-win32-slave01: 16G free
    • moz2-win32-slave02: 17G free
    • moz2-win32-slave03: 17G free
    • moz2-win32-slave05: 16.5G free
    • moz2-win32-slave06: 15.5G free
    • moz2-win32-slave11: 22G free
    • moz2-win32-slave12: 18G free
    • moz2-win32-slave13: 22G free

Build 1


pushing to ssh://
searching for changes
** unknown exception encountered, details follow
** report bug details to
** or
** Mercurial Distributed SCM (version 0.9.5)
Traceback (most recent call last):
 File "/tools/python/bin/hg", line 14, in <module>
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 20, in run
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 29, in dispatch
   return _runcatch(u, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 45, in _runcatch
   return _dispatch(ui, args)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 348, in _dispatch
   ret = _runcommand(ui, options, cmd, d)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 401, in _runcommand
   return checkargs()
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 357, in checkargs
   return cmdfunc()
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 340, in <lambda>
   d = lambda: func(ui, repo, *args, **cmdoptions)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 2147, in push
   r = repo.push(other, opts['force'], revs=revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 1385, in push
   return self.push_unbundle(remote, force, revs)
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 1465, in push_unbundle
   return remote.unbundle(cg, remote_heads, 'push')
 File "/tools/python/lib/python2.5/site-packages/mercurial/", line 195, in unbundle
   l = int(self.pipei.readline())
ValueError: invalid literal for int() with base 10: 
remote: abort: could not lock repository /repo/hg/mozilla/l10n-central/be:  Permission denied
program finished with exit code 1
    • Turns out that we never asked for 'ffxbld' to have push access to l10n repositories. Filed bug 458908 to get that enabled.
  • Tagging went fine after this
  • Two action items:
    • Dependent scheduler did not fire TODO bug 459213
    • Check-in comments say 'cltbld' when they should say 'ffxbld' TODO bug 459214


  • Automated, no problems.
  • Manually pushed builds to stage:
# as
cd /home/ftp/pub/firefox/nightly
mkdir -p 3.1b1-candidates/build1
cd 3.1b1-candidates/build1
mkdir linux-i686 mac win32 update source
# as cltbld@moz2-linux-slave06 (where the source tarball/bundle were generated
cd /builds/moz2_slave/source
scp firefox-3.1b1*


  • No problems with linux build
  • Created *_info.txt files on stage
  • Created en-US subdirs on stage:
cd /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1
mkdir linux-i686/en-US mac/en-US win32/en-US
  • On moz2-linux-slave05:
cd /builds/moz2_slave/linux_build/build/obj-firefox
make package
make -C tools/update-packaging
# upload
scp firefox-3.1b1.en-US.linux-i686.tar.bz2
cd update
scp firefox-*
  • Win32 and mac builds got interrupted when the Buildbot master hung because of graph server downtime. bug 458939
    • when the master came back up they were restarted
  • A mac nightly build got uploaded by mistake (bug 458968), redid on bm-xserve19 with
# the old one moved to stage:~cltbld/wrong-mac-en-US-firefox-3.1b1.dmg
cd /builds/moz2_slave/macosx_build/build/obj-firefox/ppc
make package
make -C tools/update-packaging
scp dist/firefox-3.1b1.en-US.mac.dmg
cd update
scp firefox-*
  • Uploaded win32 build
cd /e/builds/moz2_slave/win32_build/build/obj/dist
scp firefox-*
cd install/sea
scp firefox-* " Setup 3.1 Beta 1"
cd ../../update
scp firefox-*


  • Used slightly modified versions of this script to do l10n: (had to locally modify because of os-specific problems)
  • trimmed ja-JP-mac out on linux & win32, trimmed ja out on mac
  • should have 35 locales per platform - verified this with some bash-fu
  • This is how I pushed builds - with some extra munging depending on platform:
cd /builds/releases/3.1b1/mozilla-central/obj/dist
mkdir stage
cd stage
mkdir update
cp ../*.bz2 .
cp ../install/*.mar update/
# downloaded rename script from here:
perl ../ .
cd update
perl ../../ .
# had to do some special munging on mac and win32 because it doesn't name the 'b1' part properly
cd linux-i686
mkdir xpi
cd xpi
cp ../../../install/*.xpi .
for i in *.xpi; do locale=`echo $i | cut -f3 -d.`; mv $i $locale.xpi; done
cd ../../
# now in 'stage' directory again
rsync -av . # append 'unsigned/' for win32



  • Had to locally patch to support betas. bug 408958
  • Noticed that checksum-files also generates hashes for unsigned things FIXME bug 459215
    • Manually removed all 'unsigned' entries from the *SUMS files (because they will be trimmed before we release)


No need to run l10nverify because this is the first release we're shipping locales on.

Generate Updates

  • FIXME - hg tools repo should be tagged with UPDATE_PACKAGING tag
    • Tagged hg.m.o/build/tools with UPDATE_PACKAGING_R6 for future use
  • FIXME - ReleaseUpdatesFactory needs to properly handle the '-u' flag to bumping script bug 459216
  • FIXME - patch in bug 454261 needs to be landed in m-c
cvs co -d patcher -r UPDATE_PACKAGING_R6 mozilla/tools/patcher
cvs co -d MozBuild -r UPDATE_PACKAGING_R6 mozilla/tools/release/MozBuild
cvs co -d Bootstrap -r UPDATE_PACKAGING_R6 mozilla/tools/release/Bootstrap/
cvs co -d patcher-configs mozilla/tools/patcher-configs
hg clone
wget -O shipped-locales
perl tools/release/ -p firefox -v 3.1b1 -a 3.1b1 -o 3.1a2 -b 1 -c patcher-configs/moz191-branch-patcher2.cfg -t -f -d -l shipped-locales
# edited patcher config file with a few things:
#  * make <partial> url the same as <complete> url
#  * make 3.1a2 completemarurl the same as 3.1b1 completemarurl (because patcher won't download anything if it can't download the previous builds MARs) - TODO - revert this after snippets are done
#  * delete empty 'past-update' line
cd temp/firefox
ln -s 3.1b1 3.1a2-3.1b1
# manually had to move en-US builds to "proper" location otherwise --create-patchinfo will not work
cd 3.1a2-3.1b1/ftp/firefox/nightly/3.1b1-candidates/build1/update/linux-i686/en-US
cp ../../../../../../../firefox-3.1b1.en-US.linux-i686.complete.mar firefox-3.1b1.complete.mar
cd ../../mac/en-US
cp ../../../../../../../firefox-3.1b1.en-US.mac.complete.mar firefox-3.1b1.complete.mar
cd ../../win32/en-US
cp ../../../../../../../firefox-3.1b1.en-US.win32.complete.mar firefox-3.1b1.complete.mar
cd ../../../../../../../../../../../
perl --create-patchinfo --partial-patchlist-file=patchlist.cfg  --app=firefox --config=patcher-configs/moz191-branch-patcher2.cfg
# push out snippets
cd temp/firefox/3.1a2-3.1b1
rsync -av -e 'ssh -i ~/.ssh/aus' aus2.test/
rsync -av -e 'ssh -i ~/.ssh/aus' aus2/
# checked-in the patcher-config used for this release (version 1.3, moz191-branch-patcher2.cfg)
# also checked-in the version to use going forward, which reverts the 'completemarurl' change for 3.1a2

Publish Updates to Test Channels (betatest)

cd /opt/aus2/snippets/staging
~/bin/backupsnip 20081008-Firefox-3.1b1 # typo'ed, too late to fix it
~/bin/pushsnip 20081008-Firefox-3.1b1-test

Update Verify

Note that update verify only tests old versions, so only en-US was tested:

bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./ -c moz191-firefox-mac.cfg       
Using config file moz191-firefox-mac.cfg
07:29:15 URL: [16903884/16903884] -> "update/partial.mar" [1]
07:29:16 URL: [16903884/16903884] -> "update/complete.mar" [1]
07:29:17 URL: [16903884/16903884] -> "update/partial.mar" [1]
07:29:18 URL: [16903884/16903884] -> "update/complete.mar" [1]
bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./ -c moz191-firefox-win32.cfg
Using config file moz191-firefox-win32.cfg
07:29:30 URL: [10307200/10307200] -> "update/partial.mar" [1]
07:29:31 URL: [10307200/10307200] -> "update/complete.mar" [1]
07:29:34 URL: [10307200/10307200] -> "update/partial.mar" [1]
07:29:35 URL: [10307200/10307200] -> "update/complete.mar" [1]
bm-xserve01:/builds/releases/3.1b1/update_verify/updates cltbld$ ./ -c moz191-firefox-linux.cfg
Using config file moz191-firefox-linux.cfg
07:29:40 URL: [9610314/9610314] -> "update/partial.mar" [1]
07:29:41 URL: [9610314/9610314] -> "update/complete.mar" [1]
07:29:43 URL: [9610314/9610314] -> "update/partial.mar" [1]
07:29:44 URL: [9610314/9610314] -> "update/complete.mar" [1]

Update Bouncer

Manually added entries to

zh-CN build2

  • Localizer landed fix on the release branch:
  • Tagged that changeset as FIREFOX_3_1b1_BUILD2 and FIREFOX_3_1b1_RELEASE
  • re-generated zh-CN builds on all platforms (no logs, sorry).
  • Pushed them to
  • Symlinked everything but signed win32 builds in (those will be done post zh-CN build2 signing):
1087  for i in `ls ../../build1/linux-i686`; do ln -s ../../build1/linux-i686/$i; done
for i in `ls ../../../build1/linux-i686`; do ln -s ../../../build1/linux-i686/$i; done
for i in `ls ../../build1/mac`; do ln -s ../../build1/mac/$i; done
for i in `ls ../../../build1/mac`; do ln -s ../../../build1/mac/$i; done
for i in `ls ../../../../build1/unsigned/update/win32`; do ln -s ../../../../build1/unsigned/update/win32/$i; done
for i in `ls ../../../build1/unsigned/win32`; do ln -s ../../../build1/unsigned/win32/$i; done
for i in `ls ../../../../build1/unsigned/win32/xpi`; do ln -s ../../../../build1/unsigned/win32/xpi/$i; done
history | grep for
sign zh-CN build2
  • Followed instructions with these tweaks:
  • Only pulled win32 en-US (build1) and linux/mac/win32 zh-CN (build2) into unsigned-build2:
rsync -av -e ssh  --include=linux-i686/ --include=mac/ --include=unsigned/ --include=*irefox* --include=win32  --include=*zh-CN* --include=update/ --exclude=*${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ unsigned-build${BUILD}
rsync -av -e ssh --include=unsigned/ --include=*irefox* --include=win32  --include=*en-US* --include=update/ --exclude=*${PRODUCT}/nightly/${VERSION}-candidates/build1/ unsigned-build${BUILD}
  • Ran in previously signed mode (-p)
  • After running trimmed out en-US builds from signed-build2 before generating detached sigs and pushing back (because there's no need to do either of these for en-US on build2)
  • Pushed builds back
  • Regenrated MD5SUMS and SHA1SUMS (as per instructions)
    • Had to add '-follow' to ~/bin/checksum-files:sub gather_files() to make sure non-zh-CN files got sums generated - reverted that change afterwards

Push to mirrors

This is a bit more complicated than usual because we did a respin for zh-CN. Did the following before pushing live:

# cltbld@stage
cd /data/cltbld
# copy everything from build1
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned* /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/ firefox-3.1b1/
# copy only zh-CN from build2
rsync -av --include=win32 --include=mac --include=linux-i686 --include=[Ff]irefox* --include=*zh-CN* --exclude=* /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/ firefox-3.1b1/
cd firefox-3.1b1
-bash-3.00$ md5sum mac/ga-IE/Firefox\ 3.1\ Beta\ 1.dmg
610cb5adb38d76fa251eda8a777a5d49  mac/ga-IE/Firefox 3.1 Beta 1.dmg
-bash-3.00$ md5sum /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/ga-IE/Firefox\ 3.1\ Beta\ 1.dmg
610cb5adb38d76fa251eda8a777a5d49  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build1/mac/ga-IE/Firefox 3.1 Beta 1.dmg
-bash-3.00$ for i in linux-i686 mac win32; do md5sum $i/zh-CN/[Ff]irefox* | grep -v asc; md5sum /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/$i/zh-CN/[Ff]irefox* | grep -v asc; done
c7f357cc3c43e4310fba7381f565a593  linux-i686/zh-CN/firefox-3.1b1.tar.bz2
c7f357cc3c43e4310fba7381f565a593  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/linux-i686/zh-CN/firefox-3.1b1.tar.bz2
5d41dea708e0a9d85990e85f84a4958a  mac/zh-CN/Firefox 3.1 Beta 1.dmg
5d41dea708e0a9d85990e85f84a4958a  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/mac/zh-CN/Firefox 3.1 Beta 1.dmg
58b6c8f5bcf83c4cb80e5b7d727f8930  win32/zh-CN/Firefox Setup 3.1 Beta 1.exe
58b6c8f5bcf83c4cb80e5b7d727f8930  /home/ftp/pub/firefox/nightly/3.1b1-candidates/build2/win32/zh-CN/Firefox Setup 3.1 Beta 1.exe

And to actually push to mirrors:

rsync -av /data/cltbld/firefox-3.1b1/ /home/ftp/pub/firefox/releases/3.1b1/
# copy the KEY file

Updated /pub/ with 3.1b1

Create and push releasetest channel snippets

We needed to manually create releasetest channel snippets in order for QA to be able to test bouncer links. This was done with the following commands:

cd ~
rsync -av /opt/aus2/snippets/staging/20081008-Firefox-3.1b1/ 20081008-Firefox-3.1b1-releasetest/
cd 20081008-Firefox-3.1b1-releasetest
for i in `find . | grep 'en-US$'`; do cd $i; mv beta releasetest; cd ../../../../../; done
cd ~
rsync -av 20081008-Firefox-3.1b1-releasetest/ /opt/aus2/snippets/staging/20081008-Firefox-3.1b1-releasestest/
# push the test snippets
~/bin/pushsnip 20081008-Firefox-3.1b1-releasetest

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz191-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd 20081008-Firefox-3.1b1
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20081008-Firefox-3.1b1-releasetest/$a");'

Publish Updates to Beta Channel

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20081008-Firefox-3.1b1
~/bin/pushsnip 20081008-Firefox-3.1b1

QA found a problem with Mac 3.1a1 updates. When I generated the 'releasetest' snippets I accidentally renamed the mac 3.1a1 'beta' dir to 'releasetest'. Fixed in the '20081008-Firefox-3.1b1' dir, and pushed again:

cd /opt/aus2/snippets/staging/Firefox/3.1a1/Darwin*/200810*
mv releasetest beta
cd /opt/aus2/snippets/staging
~/bin/pushsnip 20081008-Firefox-3.1b1


  • FIXME Need to create contrib/ and contrib-localized/ dirs in release structure,
cd /pub/
mkdir contrib{,-localized}
chown cltbld:firefox contrib*
chmod 2775 contrib*
  • FIXME Ownership on release tree not all set to cltbld:firefox
chown -R cltbld:firefox *
  • FIXME permissions not set uniformly to 644 on files, and 755 on dirs (except contrib*)
find . ! -perm 644 -type f -exec chmod -v 644 {} \;
find . ! -perm 755 -type d ! -name '*contrib*' -ls -exec chmod -v 755 {} \;
  • checks
find . ! -user cltbld
find . ! -group firefox
find . -type f ! -perm 644
find . -type d ! -perm 755