From MozillaWiki
Jump to: navigation, search

Build Engineers

joduinn & nthomas & bhearsum Version/config bump bug 426307

Bonsai queries

  • RC1
    • cvsroot
    • (l10n bonsai link goes here)


Firedrill based on release branch for

Module Tag Pull date
cvsroot/mozilla RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_14_RC1 FIREFOX_2_0_0_14_RELEASE 2008-04-04 XX:XX PDT
l10n/l10n RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_14_RC1 2008-04-04 XX:XX PDT

Updated CVS Tags devmo page.

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] RC1: production-pacifica-vm
[Mac compressed] RC1: bm-xserve05
[Linux compressed] RC1: production-prometheus-vm




Setup before starting:

  • Bump bootstrap.cfg & master.cfg in bug, get review, land
  • Did a 'cvs up' in /builds/buildbot/Automation/bootstrap-configs
    • All files are now Up-to-date
  • Did a 'cvs up' in /builds/buildbot/Automation/buildbot-configs
    • master.cfg is "Locally Modified" to contain passwords. Identical to repository version otherwise.
    • Turned off the nightly builders on production-1.8-master to make sure they don't interfere with the release. (We need to figure out how to avoid this in the future).
    • update mozilla1.8 tinderbox to say "no nightlies right now"
    • Ran 'buildbot reconfig'
  • On production-1.8-master, make sure updates from previous release are present. Today, we verified the contents in /home/ftp/pub/firefox/nightly/
  • update /builds/tinderbox/mozilla/tools/tinderbox to the Bootstrap tag (need to revert this before restarting nightlies)
  • ensure that machines have enough resources
    • production-1.8-master
      • removed (but leave the dirs so we don't have to set them up with the right perms again)
        • /data/cltbld/thunderbird-
        • /home/ftp/pub/thunderbird/nightly/*candidates
        • /home/ftp/pub/thunderbird/nightly/2008-*
        • /home/ftp/pub/thunderbird/nightly/2008/03/*
        • /home/ftp/pub/thunderbird/nightly/2008/04/*
      • Did not remove /build/tags -- plenty of space on /builds
    • On bm-xserve05 remove the contents of /builds/verify/*
    • On production-prometheus-vm, checked /builds/updates for really old releases
    • Free space on production-1.8-master before starting: /builds - 26G, /data - 26G
    • Free space on bm-xserve05 before starting: / - 36G
    • Free space on production-prometheus-vm before starting: /builds - 28G
    • Free space on production-pacifica-vm before starting: C - 36G
  • Kicked off the automation:
buildbot sendchange --username=nthomas --master=localhost:9989 -m"Firefox RC1" release


  • automated - nothing to do


  • automated - nothing to do


  • automated - nothing to do


  • Signing doc
  • Note: clear space on production-1.8-master so that later stage step does not run out of space - wasn't required, had nearly 8GB free after Stage (started with 28GB free on /data)
  • slave on production-1.8-master had dropped, killing the first sign step as it waited for the signing log
  • Some partitions on the master had gone read-only (bug 407796) and a simple reboot left the buildbot master unable to start. Ran fsck on each partition and that resolved it. Editing master.cfg to start doing work at stage


  • automated - nothing to do
  • metadiff is OK (empty, as normal for Fx2)

Generate Updates

  • automated - no problems

Publish Updates to Test Channels (betatest & releasetest)

  • backupsnip took longer than the default timeout of 1 hour for a Shell() call, and got killed. Reached into the slave to add a 7200 second timeout to the call, modified master.cfg to make Push the first (uncommented) step of the updateFactory, reconfig'd the master and sent a change
  • worked fine, 37 minutes to backup
  • Snippet dirs have prefix 20080407-Firefox-

Update Verify

  • automated - no problems
  • logs check out fine


  • automated - no problems

Push Updates to Beta Channels

While waiting for QA to give "ok", on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ time ~/bin/backupsnip 20080407-Firefox-
 real    39m51.184s
 user    0m23.217s
 sys     0m33.234s

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update mozilla/tools/release/bin/ 
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080407-Firefox-
 real    2m41.935s
 user    0m0.107s
 sys     0m9.752s

Sign Installers

  • Done manually using these installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually added entries to

Push to mirrors

  • get the last formal "go" from QA/Dev/website/IT/release-drivers
  • push the stage-merged directory to the releases area:
# on stage (as cltbld)
rsync -av /data/cltbld/firefox- /home/ftp/pub/firefox/releases/

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20080407-Firefox-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080407-Firefox-$a");'

Publish Updates to Release Channel

  • In case you need to cancel/abort updates, the quickest way to do this is here
  • After QA finished testing the releasetest channel, enabled release channel:
 # on aus2-staging
 # put snippets on release/live channel
 $ cd /opt/aus2/snippets/staging
 $ sudo su - cltbld
 $ ~/bin/pushsnip 20080407-Firefox-
  • VERIFYFIX: Had to chmod +x ~/bin/pushsnip


  • Move forward the "latest" and "latest-2.0" symlinks:
 cd /home/ftp/pub/firefox/releases/
 rm latest
 ln -s latest
 rm latest-2.0
 ln -s latest-2.0
  • Update stage:/etc/rsyncd-mozilla-current.exclude

Free Software Builds

  • Bump the BuildTag definition in the three, like this
  • Done on production machines:
    • production-pacifica-vm (win32)
    • bm-xserve05 (macosx)
    • production-prometheus-vm (linux)
  • Start builds:
cd /builds/tinderbox/Fx-Mozilla1.8-FS
./ --once --depend --config-cvsup-dir `pwd`/tinderbox-configs/ 2>&1 | tee FIREFOX_2_0_0_14_FS.log
  • Stage
# as cltbld@stage
mkdir -p ~/firefox-
cd ~/firefox-
# rsync from push dirs
rsync -av /home/ftp/pub/firefox/nightly/2008-04-16-04-mozilla1.8-fs/ ./prestage/
rsync -av /home/ftp/pub/firefox/nightly/2008-04-15-14-mozilla1.8-fs/ ./prestage/
rsync -av prestage/ prestage-trimmed/
rm -v prestage-trimmed/*.zip
rsync -av prestage-trimmed/ stage/
cd stage
~/bin/groom-files --long="" .
chown -R cltbld:firefox * .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ../
rsync -av stage/ /home/ftp/pub/firefox/releases/