From MozillaWiki
Jump to: navigation, search

Sections in italics have not been completed yet.

Build Engineers

Version/config bump/tracking in bug 547309

Bonsai queries

Build 1: cvsroot


Build 1:

Module Parent Branch Created Tag/Branch Pull date
cvsroot/mozilla MOZILLA_1_8_BRANCH GECKO181_20100228_RELBRANCH 2010-02-26 08:40 PST
GECKO181_20100228_RELBRANCH THUNDERBIRD_2_0_0_24_RELEASE THUNDERBIRD_2_0_0_24_BUILD1 2010-02-28 14:22 PST
l10n/l10n MOZILLA_1_8_BRANCH GECKO181_20100228_RELBRANCH 2009-08-12 08:29 PDT
GECKO181_20100228_RELBRANCH THUNDERBIRD_2_0_0_24_RELEASE THUNDERBIRD_2_0_0_24_BUILD1 2010-02-28 14:48 PST

Build data

Type Build machine
[Windows installer/zip] production-patrocles
[Mac compressed] bm-xserve05
[Linux compressed] production-crazyhorse


Build 1


Slave cleanup

  • production-prometheus-vm
    • need ~10GB for tag, source, update generation and linux verify, stage;
    • removed /builds/tags/THUNDERBIRD_2_0_0_2{1,2}_BUILD1, source/thunderbird-{1,2}, updates/thunderbird-, verify/thunderbird-{1,2}
    • 16G free
  • production-crazyhorse
    • 14GB free on /builds - t'box will remove existing linux build so that's fine
  • bm-xserve05
    • need ~20GB for mac builds, l10n_verify, mac update verify
    • removed /builds/verify/*, /builds/tinderbox/Fx-Mozilla1.8-Nightly/Darwin_8.7.0_Depend, /builds/tinderbox/Fx-Mozilla1.8-l10n-release/Darwin_8.7.0_Depend, /builds/tinderbox/Fx-Mozilla1.8-release/Darwin_8.7.0_Depend
    • 27G free at start
  • production-patrocles
    • 41G free on e:, t'box remove existing win32 build so that's fine
  • production-pacifica-vm
    • 29G free on c:, plenty for just doing update verify

NB: Dep. builds fire every 2 hours and will compete for release jobs. Might be worth disabling them. Timing was OK so didn't bother.

Setup master before starting:

  • update mozilla/tools/buildbot-configs/automation/production/tb-master.cfg for most recent milestone of bootstrap (RELEASE_AUTOMATION_M15_1)
  • as buildmaster@production-1.8-master, cd /builds/buildbot/Thunderbird-Automation/buildbot-configs. Check for local diff, revert any changes from last release, dry run a cvs up, then cvs up
  • master.cfg and tb-master.cfg are "Locally Modified" to contain passwords; identical to repository version otherwise.
  • Ran 'buildbot reconfig /builds/buildbot/Thunderbird-Automation'
  • Buildbot waterfall is here (MPT-VPN); this is a separate instance from the (now defunct) Firefox 2.0.0.x master, with production-prometheus-vm and bm-xserve05 running a slave for each master

Login to production-1.8-master as cltbld and start automation run:

buildbot sendchange --username=nthomas --master=localhost:9990 -m"Thunderbird build 1" release


No problems


No problems


Linux and Mac: no problems

Win32 failure 1: build died publishing symbols to symbol server:

log: Running shell command in /builds/symbols/thunderbird-
log:   arg0: tools/
log:   arg1:
log: Starting time is 16:03:09 02/28/10
log: Logging output to /builds/release/logs/build_THUNDERBIRD_2_0_0_24_BUILD1-symbols.log
log: Timeout: 600
log: output: Transferring symbols...
scp: /mnt/netapp/breakpad/symbols_tbrd/: Permission denied

Step Build died: shell call returned bad exit code: 1 at Bootstrap/ line 110.


mkdir fix-tags; cd $_;
cvs co -r MOZILLA_1_8_BRANCH_release -d MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/thunderbird/win32
cvs tag -d THUNDERBIRD_2_0_0_24_RELEASE
cvs tag -d THUNDERBIRD_2_0_0_24_RELEASE_l10n
cvs tag -d THUNDERBIRD_2_0_0_24_BUILD1
cvs tag -d THUNDERBIRD_2_0_0_24_BUILD1_l10n
  • retrigger build
  • remove thunderbird/nightly/2010/02/2010-02-28-15-thunderbird2.0.0.24/ and symlink

Win32 failure 2: Failed again when uploading win32_info.txt to ftp - it read the directory for the first build out of the build log because Bootstrap appends by default. Fix

  • untag config files as above
  • on production-patrocles, delete /builds/release/logs/*24*
  • retrigger build
  • remove thunderbird/nightly/2010/02/2010-02-28-17-thunderbird2.0.0.24 and symlink

Win32 failure 3: Had not removed a symbol directory and Bootstrap complained. Lets try this one more time, because we're all having fun.

  • untag config files as above
  • on production-patrocles, delete
    • /builds/release/logs/*24*
    • /builds/symbols/thunderbird-
    • /builds/configs/thunderbird-
  • retrigger build
  • remove thunderbird/nightly/2010/02/2010-02-28-19-thunderbird2.0.0.24 and symlink
  • Finally succeeds, huzzah!


  • Signing doc
  • NB: Doc change - do only the linked section.
  • No problems

l10n verification

No problems.

Generate updates

The patcher config bump set


when we'd previously used Thinking this was in error I modified the config while patcher was downloading the complete mars. However this is bad form, and it turns out that we redirect from


so the change was fine.

Regenerate the snippets to avoid needing a redirect

# cltbld@production-prometheus-vm
cd /builds/updates/thunderbird-
cvs up -C moz18-branch-patcher2.cfg
# cvs stat gives rev 1.30 unmodified
cd ../patcher/temp/thunderbird/
mkdir v1-snippets
mv aus2* v1-snippets

cd /builds/updates/thunderbird-
./ --create-patchinfo --app=thunderbird --config=../config/moz18-branch-patcher2.cfg

cd /builds/updates/thunderbird-
diff -r -I '^detailsUrl' v1-snippets/aus2      aus2
diff -r -I '^detailsUrl' v1-snippets/aus2.beta aus2.beta
diff -r -I '^detailsUrl' v1-snippets/aus2.test aus2.test
# no output for all three diffs, spot check comfirms expected change in detailsUrl
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2/
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.beta/
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.test/

# cltbld@aus-staging
~/bin/backupsnip 20100302-Thunderbird-
~/bin/pushsnip   20100302-Thunderbird-

Publish updates to test channels (betatest & releasetest)

No problems.

Update verify

No problems.


No problems.

Push updates to beta channel

Turns out we should be using the snippets that point to Before go

# cltbld@aus2-staging
~/bin/backupsnip 20100301-Thunderbird-

After go

~/bin/pushsnip   20100301-Thunderbird-

Landed the modification to moz18-branch-patcher2.cfg.

Sign installers

Done manually using these installer-signing-instructions here. NB: Doc location change, content unmodified.

  • complete stage-merged:
# cltbld@stage
cd /data/cltbld/thunderbird-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
cd /data/cltbld/thunderbird-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:thunderbird .
chmod 644 *SUMS

Update Bouncer

Done. Left out Solaris because they have something unusual for and I want to hear directly what should be configured.

Push to mirrors

# on stage (as cltbld)
rsync -av /data/cltbld/thunderbird- /home/ftp/pub/thunderbird/releases/

Update rsync module

Thunderbird 2.0.0.x is no longer in the mozilla-current rsync module. Nothing to do here.

Final verification

  • Verify that releasetest points to valid bouncer links:
hg clone
cd tools/release/updates
cat moz18-thunderbird-{win32,mac,linux}.cfg | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20100301-Thunderbird-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20100301-Thunderbird-$a");'

Publish updates to release channel

  • In case you need to cancel/abort updates, the quickest way to do this is here
  • While waiting for QA to finish testing, did a backup:
# on aus2-staging verify we have correct tools
sudo su - cltbld
cd ~/bin
cvs update -dP
~/bin/backupsnip 20100301-Thunderbird-
  • Discovered that 20100301-Thunderbird- hadn't been pushed to fix the detailsURL for betatest and releasetest, which QA then caught when the MoMo url was a 404
~/bin/backupsnip 20100301-Thunderbird-
~/bin/pushsnip   20100301-Thunderbird-

~/bin/backupsnip 20100301-Thunderbird-
  • After QA finished testing the releasetest channel and go from release driver, enable release channel:
# on aus2-staging
sudo su - cltbld
~/bin/pushsnip 20100301-Thunderbird-

Update symlink

# cltbld@stage
cd /pub/
rm latest-2.0
ln -s latest-2.0