From MozillaWiki
Jump to: navigation, search

Build Engineers

Version/config bump bug

Bonsai queries

Build 1:


Updated CVS Tags devmo page.

Build 1:

Module Branch Tag Pull date
cvsroot/mozilla MOZILLA_1_8_BRANCH FIREFOX_2_0_0_18_BUILD1 2008-10-29 10:45 PDT
l10n/l10n MOZILLA_1_8_BRANCH FIREFOX_2_0_0_18_BUILD1 2008-10-29 10:45 PDT

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] production-pacifica-vm
[Mac compressed] bm-xserve05
[Linux compressed] production-prometheus-vm


Build 1

Slave cleanup:

  • production-prometheus-vm
    • /builds/verify/firefox-, /builds/verify/thunderbird-
    • /builds/updates/firefox-, /builds/updates/thunderbird-
    • /build/source/firefox-, /builds/source/thunderbird-
    • /data/cltbld/firefox-, /data/cltbld/thunderbird-
    • /builds/tags/*
  • bm-xserve05
    • /builds/verify/firefox-, /builds/verify/thunderbird-
  • production-pacifica-vm
    • removed twistd.log.?? and twistd.log.???
  • production-1.8-master
    • removed /builds/buildbot/Automation/twistd.log.???

Bootstrap tag: RELEASE_AUTOMATION_M11 (unchanged from Fx2.0.0.17)

Setup before starting:

  • Updated /builds/buildbot/Automation/buildbot-configs on production-1.8-master.
    • master.cfg is "Locally Modified" to contain passwords. Identical to repository version otherwise.
    • Turned off the nightly builders on production-1.8-master to make sure they don't interfere with the release. (We need to figure out how to avoid this in the future).
      • commented out "c['schedulers'].append(depend_scheduler)"
    • updated mozilla1.8 tinderbox to say "no nightlies right now"
  • ensure that machines have enough resources
    • production-prometheus-vm - 31G in /
    • production-1.8-master - 31G in /builds, 19G in /data
    • production-pacifica-vm - 29G in c:/
    • bm-xserve05 - 19G in /

Issued sendchange:

  • buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox build" release


Automated, no problems.


Automated, no problems.


Automated, no problems.


Automated, no problems.


  • freed up space on keymaster, removed builds from:
    • firefox-
    • firefox-3.0.1
    • firefox-3.1a1
    • firefox-3.1a2
    • thunderbird-
    • thunderbird-3.0a2
  • Signing doc


Automated, no problems.

Generate Updates

Automated, no problems.

Publish Updates to Test Channels (betatest & releasetest)

Automated, no problems.

Push updates to beta channel

Ran the pushsnip commands for both and 3.0.4 at the same time on aus2-staging.

# put snippets on beta
$ sudo su - cltbld
# make sure using latest version of scripts in mozilla/tools/release/bin/
$ cd bin
$ cvs update . 
$ cd /opt/aus2/snippets/staging
# note the required parameter must match what will be used with pushsnip below. 
$ time ~/bin/backupsnip 20081103-Firefox-3.0.4-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20081105-1-pre-20081103-Firefox-3.0.4-beta.tar.bz2 .

real	24m5.278s
user	0m24.322s
sys	0m27.792s

$ time ~/bin/pushsnip 20081030-Firefox-
real	1m46.869s
user	0m0.205s
sys	0m7.835s

$ time ~/bin/pushsnip 20081103-Firefox-3.0.4-beta
real	0m50.238s
user	0m0.096s
sys	0m3.560s

Sign Installers

Done manually using these installer-signing-instructions here

  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually added entries to
  • Note for next release: Do not remove the Check Now bit on the Firefox- Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox- /home/ftp/pub/firefox/releases/

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd 20081030-Firefox-
time find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20081030-Firefox-$a");'

real	0m43.382s
user	0m0.776s
sys	0m13.461s

Encountered 404s when trying to retrieve files from two mirrors:


justdave indicated that this was a known issue with sentry adding mirrors before they have all the files. I was able to access all the 404 files on subsequent attempts.

Publish Updates to Release Channel

backupsnip was run as part of the Firefox 3.0.4 release process: 3.0.4 build notes

# on aus2-staging
$ time ~/bin/pushsnip  20081030-Firefox-
real	0m55.028s
user	0m0.131s
sys	0m6.420s


  • On stage.m.o, do the following:
    • Update "latest" symlinks
$ cd /pub/
$ mv latest-2.0 latest-2.0.old
$ ln -s latest-2.0
$ rm latest-2.0.old