From MozillaWiki
Jump to: navigation, search

Please be sure to use

  • RELEASE_AUTOMATION_M15 in the buildbot master config (for bug 496491)

Build Engineers

Version/config bump/tracking in bug 506948

Bonsai queries

Build 1: cvsroot


Build 1:

Module Parent Branch Created Tag/Branch Pull date
cvsroot/mozilla MOZILLA_1_8_BRANCH GECKO181_20090812_RELBRANCH 2009-08-12 08:29 PDT
GECKO181_20090812_RELBRANCH THUNDERBIRD_2_0_0_23_RELEASE THUNDERBIRD_2_0_0_23_BUILD1 2009-08-12 08:29 PDT
l10n/l10n MOZILLA_1_8_BRANCH GECKO181_20090812_RELBRANCH 2009-08-12 08:29 PDT
GECKO181_20090812_RELBRANCH THUNDERBIRD_2_0_0_23_RELEASE THUNDERBIRD_2_0_0_23_BUILD1 2009-08-12 08:29 PDT

Build data

Type Build machine
[Windows installer/zip] production-patrocles
[Mac compressed] bm-xserve05
[Linux compressed] production-crazyhorse


Build 1


Slave cleanup

  • production-prometheus-vm
    • need ~10GB for tag, source, update generation and linux verify, stage; 20G free to begin with - didn't do any cleanup
  • production-crazyhorse
    • already had 15GB free on /builds - should be fine
  • bm-xserve05
    • need ~20GB for mac builds, l10n_verify, mac update verify
    • removed /builds/verify/*, /builds/tinderbox/Fx-Mozilla1.8-Nightly/Darwin_8.7.0_Depend, /builds/tinderbox/Fx-Mozilla1.8-l10n-release/Darwin_8.7.0_Depend, /builds/tinderbox/Fx-Mozilla1.8-release/Darwin_8.7.0_Depend
    • 33G free at start
  • production-patrocles
    • driveE: already had 43GB free
  • production-pacifica-vm
    • already 29.6GB free, plenty for update verify
  • Landed and tagged updated bootstrap config

Setup master before starting:

  • update mozilla/tools/buildbot-configs/automation/production/tb-master.cfg for most recent milestone of bootstrap
  • as buildmaster@production-1.8-master, cd /builds/buildbot/Thunderbird-Automation/buildbot-configs. Check for local diff, revert any changes from last release, dry run a cvs up, then cvs up
  • master.cfg and tb-master.cfg is "Locally Modified" to contain passwords; identical to repository version otherwise.
  • Ran 'buildbot reconfig /home/buildmaster/Thunderbird-Automation'
  • Buildbot waterfall is here (MPT-VPN); this is a separate instance from the Firefox master, with production-prometheus-vm and bm-xserve05 running a slave for each master

Login to production-1.8-master as cltbld and start automation run:

buildbot sendchange --username=bhearsum --master=localhost:9990 -m"Thunderbird build 1" release


  • Tag failed out with:
None of ^WIN32_MODULE_PRODUCTVERSION_STRING=$ found in file /builds/tags/THUNDERBIRD_2_0_0_23_BUILD1/cvsroot/mozilla/mail/app/module.ver: Inappropriate ioctl for device at /builds/tags/THUNDERBIRD_2_0_0_23_BUILD1/tools/release/ line 142.
    • This is because Tb nightlies were never version bumped to after
  • Checked in a version bump, updated and retaged bootstrap config
  • Had to undo the branching to get Bootstrap to retag:
cvs rtag -B -d GECKO181_20090812_RELBRANCH mozilla/
  • Restarted automation with another sendchange
  • No other problems


No problems


  • Linux: no problems
  • Mac: no problems
  • Win32: no problems


  • Signing doc
  • NB: Doc change - do only the linked section.
  • Used new signing scripts - no problems

l10n verification

  • Failed to mount or rsync any mac dmgs
    • bug 499506 removed our ability to mount DMGs with a EULA. Backed it out temporarily to get us going again, long term fixed being tracked in bug 510208
  • Nothing in l10nverify log == PASS.

Generate updates

  • Died when trying to download the mac ja-JP-mac build because it doesn't exist. The Repack log shows this error:
mkdir -p pkg-dmg.19209.9bTq6AWz/stage/.background
rsync -a --copy-unsafe-links /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/dist/branding/dsstore pkg-dmg.19209.9bTq6AWz/stage/.DS_Store
rsync -a --copy-unsafe-links /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/dist/branding/background.png pkg-dmg.19209.9bTq6AWz/stage/.background
rsync -a --copy-unsafe-links /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/dist/branding/disk.icns pkg-dmg.19209.9bTq6AWz/stage/.VolumeIcon.icns
/Developer/Tools/SetFile -a C pkg-dmg.19209.9bTq6AWz/stage
chmod -R a+rX,a-st,u+w,go-w pkg-dmg.19209.9bTq6AWz/stage
hdiutil makehybrid -hfs -hfs-volume-name Thunderbird -hfs-openfolder pkg-dmg.19209.9bTq6AWz/stage -ov pkg-dmg.19209.9bTq6AWz/stage -o pkg-dmg.19209.9bTq6AWz/hybrid.dmg
/builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/build/package/mac_osx/pkg-dmg: exited on signal 11
/builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/build/package/mac_osx/pkg-dmg: hdiutil makehybrid failed (cleaning up)
Creating hybrid image...
make[2]: *** [repackage-zip] Error 1
make[1]: *** [repackage-zip-ja-JP-mac] Error 2
make: *** [installers-ja-JP-mac] Error 2
    • Fixed with the following:
# pulled this line from the log
make -C /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/mail/locales installers-ja-JP-mac ZIP_IN=/builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/thunderbird.dmg
# don't know why this didn't exist already, but the Makefiles need it here :S
cd /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/l10n-stage
mkdir -p host/bin
cp ../host/bin/* .
make -C /builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/tools/update-packaging full-update STAGE_DIR=/builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/dist/install/2009-08-12-12-thunderbird2.0.0.23-l10n/build.25223 DIST=/builds/tinderbox/Tb-Mozilla1.8-l10n-Release/Darwin_8.7.0_Depend/mozilla/dist/l10n-stage AB_CD=ja-JP-mac
# and for the upload
scp *ja-JP-mac* install/2009-08-12-12-thunderbird2.0.0.23-l10n/build.25223/thunderbird-
scp install/*ja-JP-mac*
  • Removed /builds/updates/thunderbird- on production-prometheus-vm and restarted the updates step with Force Build

Publish updates to test channels (betatest & releasetest)

  • Failed when aus2-staging claimed to run out of disk space bug 510203
    • The /opt partition ran out of inodes - Aravind fixed it
  • To push through the rest of this, did the following;
# cltbld @ production-prometheus-vm
cd /builds/updates/thunderbird-
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.test/

And then, on aus2-staging,

# cltbld @ aus2-staging
# start screen, then
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20090813-Thunderbird-
~/bin/pushsnip 20090813-Thunderbird-

Update verify

  • Linux: fine
  • Win32: fine
  • Mac: fine


  • Failed first time because mac ja-JP-mac.xpi was uploaded with wrong name (due to the manual intervention. To fix, renamed it to the proper thing:
# cltbld @ stage
cd /home/ftp/pub/thunderbird/nightly/
mv thunderbird- ja-JP-mac.xpi

Then, clobbered /data/cltbld/thunderbird- on production-prometheus-vm and started Stage again.

  • Failed erroneously afterwards because it picked up the "FAIL" in the log from the first run and halted. Deleted the dir again, deleted the log too, and restarted it
  • No further issues.

Push updates to beta channel

# cltbld @ aus2-staging
# start screen, then
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20090813-Thunderbird-
~/bin/pushsnip   20090813-Thunderbird-

Sign installers

Done manually using these installer-signing-instructions here. NB: Doc location change, content unmodified.

  • Had to locally modify sign-release to point to the right keys because of bug 507660
  • complete stage-merged:
# on stage
cd /data/cltbld/thunderbird-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/thunderbird-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:thunderbird .
chmod 644 *SUMS

Update bouncer


Push to mirrors

  1. on stage (as cltbld)

rsync -av /data/cltbld/thunderbird- /home/ftp/pub/thunderbird/releases/

Update rsync module

On, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/
vi rsyncd-mozilla-current.exclude 

and replace the previous latest release (Tb2.0.0.22) with:

+ thunderbird/releases/
+ thunderbird/releases/**

Final verification

  • Verify that releasetest points to valid bouncer links:
hg clone
cd tools/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20090813-Thunderbird-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090813-Thunderbird-$a");'

Publish updates to release channel

~/bin/backupsnip 20090813-Thunderbird-

Update symlink

# cltbld @stage
cd /home/ftp/pub/thunderbird/releases
rm latest-2.0
rm latest
ln -s latest-2.0
ln -s latest-2.0 latest # futureproofed!