From MozillaWiki
Jump to: navigation, search

Build Engineers


Version/config bump/tracking in bug 496271

Bonsai queries

Build 1: cvsroot


Build 1:

Module Parent Branch Created Tag/Branch Pull date
cvsroot/mozilla MOZILLA_1_8_BRANCH GECKO181_20090603_RELBRANCH 2009-06-03 11:32 PDT
GECKO181_20090603_RELBRANCH THUNDERBIRD_2_0_0_22_RELEASE, THUNDERBIRD_2_0_0_22_BUILD1 2009-06-03 11:32 PDT (but see Tag below)
l10n/l10n MOZILLA_1_8_BRANCH GECKO181_20090603_RELBRANCH 2009-06-03 11:32 PDT
GECKO181_20090603_RELBRANCH THUNDERBIRD_2_0_0_22_RELEASE, THUNDERBIRD_2_0_0_22_BUILD1 2009-06-03 11:32 PDT

Build data

Type Build machine
[Windows installer/zip] production-patrocles
[Mac compressed] bm-xserve05
[Linux compressed] production-crazyhorse


Build 1

Using RELEASE_AUTOMATION_M14, same as Fx3.0.11 but new for Tb. Leaving Tinderbox on RELEASE_AUTOMATION_M11 (don't need bug bug 413491).

Slave cleanup

  • production-prometheus-vm
    • need ~10GB for tag, source, update generation and linux verify, stage; removed /builds/updates/thunderbird- to leave 15.4GB free
  • production-crazyhorse
    • already had 22GB free on /builds
  • bm-xserve05
    • need ~20GB for mac builds, l10n_verify, mac update verify (but 13GB already allocated for builds); already had 29.6GB free.
  • production-patrocles
    • driveE: already had 68GB free
  • production-pacifica-vm
    • already 29.6GB free, plenty for update verify
  • Land changes and retag tb-moz18-bootstrap.cfg
    • cvs tag -d RELEASE_AUTOMATION_M14 tb-moz18-bootstrap.cfg # to delete the tag on the old version
    • cvs tag RELEASE_AUTOMATION_M14 tb-moz18-bootstrap.cfg # to tag the new version

Setup master before starting:

  • update mozilla/tools/buildbot-configs/automation/production/tb-master.cfg for most recent milestone of bootstrap
  • as buildmaster@production-1.8-master, cd /builds/buildbot/Thunderbird-Automation/buildbot-configs. Check for local diff, revert any changes from last release, dry run a cvs up, then cvs up
  • master.cfg and tb-master.cfg is "Locally Modified" to contain passwords; identical to repository version otherwise.
  • Ran 'buildbot reconfig /home/buildmaster/Thunderbird-Automation'
  • Buildbot waterfall is here (MPT-VPN); this is a separate instance from the Firefox master, with production-prometheus-vm and bm-xserve05 running a slave for each master
  • IGNORE THIS (no longer needed now that FF2 is EOL'd): Stop nightly builds on bm-xserve05bm-xserve02 (~cltbld $ buildbot stop /builds/slave/as cltbld, go to ~/cltbld and run "./tinderbox multi stop") and annotate Mozilla1.8 tinderbox tree to explain its absence

Login to productioion-1.8-master as cltbld and start automation run:

buildbot sendchange --username=joduinn --master=localhost:9990 -m"Thunderbird build 1" release


automated - no problems


automated - no problems


automated - no problems


  • Followed Signing doc
  • Because of long delay caused by signing of FF3.5beta99 and FF3.5beta99 respin, the "wait for signing log" timed out after 24 hours. Had to:
    • Logged into production-1.8-master as buildmaster
    • manually hack out tag/srctarball/build steps, then reconfig master
    • login to production-1.8-master as cltbld
    • start automation run:
buildbot sendchange --username=joduinn --master=localhost:9990 -m"Thunderbird build 1" release

l10n verification

automated - no problems

Generate updates

automated - no problems

Publish updates to test channels (betatest & releasetest)

automated - no problems

Update verify

  • linux was fine
  • win32, mac hit problems caused by bootstrap bug 496491. nthomas told me to remove /builds/verify/thunderbird- and retry. This worked a treat.
  • QA found that sha512 checksums do not work during testing of complete updates. TB2.0.0.22 is the first release using newer sha512 checksums, and this was broken in TB2.0.0.0...TB2.0.0.6. It works fine in TB2.0.0.9 and later. The binaries, and the mar files are all fine. On Monday, we'll regenerate the snippets using old sha1 checksums. For details see bugs attached to tracking bug 496271.

Fix update snippets

# cltbld@production-prometheus-vm
cd /builds/updates/thunderbird-
cvs up -r UPDATE_PACKAGING_R7 MozBuild/

cd temp/thunderbird/
mkdir SHA512-snippets
mv aus2* SHA512-snippets

cd /builds/updates/thunderbird-
./ --create-patchinfo --app=thunderbird --config=../config/moz18-branch-patcher2.cfg

cd temp/thunderbird/
# did some checks here, diff against SHA512 and grep for hashFunction
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2/ \
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.beta/ \
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.test/ \

# cltbld@aus2-staging
~/bin/backupsnip 20090614-Thunderbird-
~/bin/pushsnip   20090614-Thunderbird-

Update verify was green on all three platforms.

Push updates to beta channel

  • Skipping backupsnip, because it was just done by bhearsum for FF3.5rc1build1.
  • Waited for formal "go" to beta.
$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ time ~/bin/pushsnip 20090614-Thunderbird-
real    1m42.893s
user    0m0.179s
sys     0m7.826s

Sign installers

  • Done manually using these installer-signing-instructions here
  • complete stage-merged:
# on stage
cd /data/cltbld/thunderbird-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/thunderbird-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:thunderbird .
chmod 644 *SUMS

Update bouncer


Push to mirrors

  1. on stage (as cltbld)

rsync -av /data/cltbld/thunderbird- /home/ftp/pub/thunderbird/releases/

Update rsync module

On, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/
vi rsyncd-mozilla-current.exclude 

and replace the previous latest release (Tb2.0.0.21) with:

+ thunderbird/releases/
+ thunderbird/releases/**

Final verification

  • Verify that releasetest points to valid bouncer links:
hg clone
cd tools/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20090610-Thunderbird-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090610-Thunderbird-$a");'

Publish updates to release channel

  • In case you need to cancel/abort updates, the quickest way to do this is here
  • While waiting for QA to finish testing, do a backup:
# on aus2-staging verify we have correct tools
$ sudo su - cltbld
$ cd ~/bin
$ cvs update -dP
# backup snippets on release/live channel
$ time ~/bin/backupsnip 20090614-Thunderbird-
real    48m15.344s
user    0m43.540s
sys     1m12.725s
  • After QA finished testing the releasetest channel and go from release driver, enable release channel:
# on aus2-staging
# put snippets on release/live channel
$ sudo su - cltbld
$ time ~/bin/pushsnip 20090614-Thunderbird-

Update symlink

$ cd /pub/

# confirm where latest symlinks point to where before starting
$ ls -la latest*
lrwxrwxrwx  1 cltbld thunderbird 8 Mar 18 13:54 latest ->
lrwxrwxrwx  1 cltbld thunderbird 8 Mar 18 13:54 latest-2.0 ->
$ rm latest    # (note no trailing slash)
$ ln -s latest
$ rm latest-2.0   # (note no trailing slash)
$ ln -s latest-2.0
# confirm latest symlinks point to newest release
$ ls -la latest*
lrwxrwxrwx  1 cltbld mozilla 8 Jun 25 17:43 latest ->
lrwxrwxrwx  1 cltbld mozilla 8 Jun 25 17:44 latest-2.0 ->


  • Undid hackery in master.cfg after signing step.