From MozillaWiki
Jump to: navigation, search

Build Engineers

bhearsum Version/config bump bug 422122

Bonsai queries


Module Tag Pull date
cvsroot/mozilla RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_13_RC1 FIREFOX_2_0_0_13_RELEASE 2008-03-09 23:00 PDT
mofo/talkback/fullsoft RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_13_RC1 2008-03-09 23:00 PDT
l10n/l10n RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_13_RC1 2008-03-09 23:00 PDT

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] RC1: production-pacifica-vm
[Mac compressed] RC1: bm-xserve05
[Linux compressed] RC1: production-prometheus-vm




Setup before starting:

  • Did a 'cvs up' in /builds/buildbot/Automation/bootstrap-configs
    • All files are now Up-to-date
  • Did a 'cvs up' in /builds/buildbot/Automation/buildbot-configs
    • master.cfg is "Locally Modified" to contain passwords. Identical to repository version otherwise.
    • Ran 'buildbot reconfig'
  • ensure that machines have enough resources
    • production-1.8-master
      • rsync -nav /home/ftp/pub/firefox/nightly/
      • removed /home/ftp/pub/thunderbird/ after comparing to staging like above
      • Did not remove /build/tags -- plenty of space on /builds
    • On bm-xserve05 remove the contents of /builds/verify/*
    • Ran 'cd /builds/updates && rm -rf firefox- thunderbird- firefox-' on production-prometheus-vm to free up some space.
    • Free space on production-1.8-master before starting: /builds - 33G, /data - 26G
    • Free space on bm-xserve05 before starting: / - 36G
    • Free space on production-prometheus-vm before starting: /builds - 29G
    • Free space on production-pacifica-vm before starting: C - 36.4G
  • Turned off the nightly builders on production-1.8-master to make sure they don't interfere with the release. (We need to figure out how to avoid this in the future).
  • Kicked off the automation:
buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox RC1" release


No problems.


No problems.


  • OS X wouldn't start, had to restart the slave and manually kick off OS X build.


No problems.


No problems.

Updates to Test Channels (betatest & releasetest)

  • Updates died when trying to pull a build from stage-old.
    • Filed bug 422421 on getting http from stage-old fixed.
    • s/stage-old/stage/ in the bootstrap.cfg and restarted updates.
    • There was some other fall-out from this, had to blow away /builds/updates/firefox- and restart Updates.
  • Rhelmer accidently interrupted updates, ran the following by hand:
ssh cltbld@production-prometheus-vm
cd /builds/updates/firefox-
./ --create-patches --app=firefox --config=../config/moz18-branch-patcher2.cfg

update Verify

  • Linux/Win32 passed
  • Mac failed because 'ka' is in verify config when it shouldn't be. This was only caught since many old builds were removed from stage.m.o
  • Fixing verify config manually so we won't hit this again.


  • failed because of a local patch (that replace 'firefox' strings with 'thunderbird'. bug 409232
  • ran manually after cvs up -CPd, no problems.

Updates to Beta Channels

While waiting for QA to give "ok", on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update .
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ time ~/bin/backupsnip 20080312-Firefox-

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080312-Firefox-

Sign Installers

  • Done manually using these installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS


  • push the stage-merged directory to the releases area:
# on stage (as cltbld)
rsync -av /data/cltbld/firefox- /home/ftp/pub/firefox/releases/
  • Move forward the "latest" and "latest-2.0" symlinks:
 cd /home/ftp/pub/firefox/releases/
 rm latest
 ln -s latest
 rm latest-2.0
 ln -s latest-2.0
  • Update stage:/etc/rsyncd-mozilla-current.exclude

Update Bouncer

  • Add entries to

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Found a few 404s to Tested two of them and are OK now - probably had to do with only partial mirror propagation.

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080312-Firefox-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080312-Firefox-$a");'

Updates to release channel

  • After QA finished testing the releasetest channel, enabled release channel:
 # on aus2-staging
 # put snippets on release/live channel
 $ cd /opt/aus2/snippets/staging
 $ sudo su - cltbld
 $ ~/bin/pushsnip 20080312-Firefox-

Had to chmod +x ~/bin/pushsnip

Free Software Builds

  • Bump the BuildTag definition in the three, like this
  • Done on production machines:
    • production-pacifica-vm (win32)
    • bm-xserve05 (macosx)
    • production-prometheus-vm (linux)
  • Start builds:
cd /builds/tinderbox/Fx-Mozilla1.8-FS
./ --once --depend --config-cvsup-dir `pwd`/tinderbox-configs/ 2>&1 | tee FIREFOX_2_0_0_13_FS.log
  • Stage
# as cltbld@stage
mkdir -p ~/firefox-
cd ~/firefox-
# rsync from push dirs
rsync -av /home/ftp/pub/firefox/nightly/2008-03-25-04-mozilla1.8-fs/ ./prestage/
rsync -av /home/ftp/pub/firefox/nightly/2008-03-25-05-mozilla1.8-fs/ ./prestage/
rsync -av prestage/ prestage-trimmed/
rm -v prestage-trimmed/*.zip
rsync -av prestage-trimmed/ stage/
cd stage
~/bin/groom-files --long="" .
chown -R cltbld:firefox * .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ../
rsync -av stage/ /home/ftp/pub/firefox/releases/