From MozillaWiki
Jump to: navigation, search

Build Engineers

Version/config bump bug

Bonsai queries

Build 1:


Build 1:

Module Branch Tag Pull date
cvsroot/mozilla MOZILLA_1_8_BRANCH THUNDERBIRD_2_0_0_16_BUILD1

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] production-crazyhorse
[Mac compressed] bm-xserve05
[Linux compressed] production-patrocles



See for all the setup notes.

Build 1


No problems.


No problems.


  • Build on win32 & linux failed because Bootstrap couldn't be checked out (ssh key error)
    • Changed id_dsa* symlinked to point to cltbld key (they were pointing to tbirdbld). This shouldn't cause problems because other steps (symbols, stage upload) that require keys point explicitly to the tbirdbld key.
  • Failed out again because tinderbox-configs directories were checked out with the tbirdbld key.
    • Re-checked them out with the cltbld key and restarted automation.
  • Linux build hung when trying to upload Talkback symbols
    • Forgot to copy over production talkback packages. Caught win32 & mac before they hit this. Manually ran the following to upload linux symbols:
cd /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla && tar jxvf /builds/tinderbox/talkback-thunderbird-1.8-linux.tar.bz2 && cp -r fullsoft /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla
cd fullsoft
rm Makefile
cd /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/fullsoft && /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/build/autoconf/make-makefile -d ..
make -C /builds/tinderbox/Tb-Mozilla1.8-Release/Linux_2.4.18-14_Depend/mozilla/fullsoft fullcircle-push FC_BUILD=2008070711
  • Mac Build failed erroneously because 'sendmail' was set wrong in the Bootstrap config (/usr/lib/sendmail instead of /usr/sbin/sendmail)
    • Checked in a bustage fix, continued on.
  • Win32 build failed out when trying to upload talkback symbols. Re-ran as follows:
cd /cygdrive/e/builds/tinderbox/Fx-Mozilla1.8-Release
./ --once --mozconfig mozconfig --depend --config-cvsup-dir --skip-mozilla
    • Talkback symbols uploaded fine with this, but builds did not.
      • Didn't know what else to try, so started win32 from scratch.
    • Talkback symbols were fine this time, but I forgot to remove the build logs before re-running. the build failed because it grabbed the wrong buildid
      • Cleared the build logs and respun


No problems


Signing doc


Lots of output from l10nverify. Most of it was from a linux kernel bump on crazyhorse and a EULA change. The build directory also changed from Tb-Mozilla1.8-release to Tb-Mozilla1.8-Release, which caused some more output.

Generate Updates

No problems

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

  • Win32 failed out because win32_logDir was set wrong. (Was /cygdrive/e/builds/release/logs, now /builds/release/logs). Because we run update_verify on production-pacifica-vm, which doesn't have an E: drive, we can't use /cygdrive/e/..
  • Re-ran and it worked fine.


Failed out because 'cltbld' on production-prometheus-vm was not part of the 'thunderbird' group. Re-ran and it worked fine.

Push Updates to Beta Channels

-bash-3.2$ ~/bin/backupsnip 20080709-Thunderbird-
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080716-1-pre-20080709-Thunderbird- .
-bash-3.2$ ~/bin/pushsinp 20080709-Thunderbird-3.0.1-beta

Sign Installers

Done manually using these installer-signing-instructions here

  • complete stage-merged:
# on stage
cd /data/cltbld/thunderbird-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/thunderbird-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Much later (20080925) - Setting the group to firefox is wrong, changed it to thunderbird.

Update Bouncer

  • Manually added entries to

Update rsync module

On, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/
vi rsyncd-mozilla-current.exclude 

and replace the previous latest release (Tb2.0.0.16) with:

+ thunderbird/releases/
+ thunderbird/releases/**

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage (as cltbld)
rsync -av /data/cltbld/thunderbird- /home/ftp/pub/thunderbird/releases/

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20080709-Thunderbird-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080709-Thunderbird-$a");'

Publish Updates to Release Channel

  • In case you need to cancel/abort updates, the quickest way to do this is here
  • While waiting for QA to finish testing, do a backup:
 # on aus2-staging
 # verify we have correct tools
 $ sudo su - cltbld
 $ cd ~/bin
 $ cvs update -dP
 # backup snippets on release/live channel
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/backupsnip 20080709-Thunderbird-
real    30m55.519s
user    0m23.555s
sys     0m32.026s
  • After QA finished testing the releasetest channel, enabled release channel:
 # on aus2-staging
 # put snippets on release/live channel
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080709-Thunderbird-
real    1m4.116s
user    0m0.059s
sys     0m3.058s


On stage.m.o, change symlink to point to new release:

 $ cd /pub/
 # confirm where latest symlinks point to where before starting
 $ ls -la latest*
lrwxrwxrwx  1 cltbld thunderbird 8 May  1 07:04 latest ->
lrwxrwxrwx  1 root   thunderbird 8 Feb 26 09:27 latest-1.5 ->
lrwxrwxrwx  1 cltbld thunderbird 8 May  1 07:04 latest-2.0 ->
 $ rm latest    (note no trailing slash)
 $ ln -s latest
 $ rm latest-2.0    (note no trailing slash)
 $ ln -s latest-2.0
 # confirm latest symlinks point to newest release
 $ ls -la latest*
lrwxrwxrwx  1 cltbld thunderbird 8 Jul 23 19:09 latest ->
lrwxrwxrwx  1 root   thunderbird 8 Feb 26 09:27 latest-1.5 ->
lrwxrwxrwx  1 cltbld thunderbird 8 Jul 23 19:10 latest-2.0 ->