Releases/Thunderbird 2.0.0.22/BuildNotes

Build Engineers

joduinn

Version/config bump/tracking in bug 496271

Bonsai queries

Build data

Type	Build machine
[Windows installer/zip]	production-patrocles
[Mac compressed]	bm-xserve05
[Linux compressed]	production-crazyhorse

Notes

Build 1

Using RELEASE_AUTOMATION_M14, same as Fx3.0.11 but new for Tb. Leaving Tinderbox on RELEASE_AUTOMATION_M11 (don't need bug bug 413491).

Slave cleanup

production-prometheus-vm
- need ~10GB for tag, source, update generation and linux verify, stage; removed /builds/updates/thunderbird-2.0.0.21 to leave 15.4GB free

production-crazyhorse
- already had 22GB free on /builds

bm-xserve05
- need ~20GB for mac builds, l10n_verify, mac update verify (but 13GB already allocated for builds); already had 29.6GB free.

production-patrocles
- driveE: already had 68GB free

production-pacifica-vm
- already 29.6GB free, plenty for update verify

Land changes and retag tb-moz18-bootstrap.cfg
- cvs tag -d RELEASE_AUTOMATION_M14 tb-moz18-bootstrap.cfg # to delete the tag on the old version
- cvs tag RELEASE_AUTOMATION_M14 tb-moz18-bootstrap.cfg # to tag the new version

Setup master before starting:

update mozilla/tools/buildbot-configs/automation/production/tb-master.cfg for most recent milestone of bootstrap
as buildmaster@production-1.8-master, cd /builds/buildbot/Thunderbird-Automation/buildbot-configs. Check for local diff, revert any changes from last release, dry run a cvs up, then cvs up
master.cfg and tb-master.cfg is "Locally Modified" to contain passwords; identical to repository version otherwise.
Ran 'buildbot reconfig /home/buildmaster/Thunderbird-Automation'
Buildbot waterfall is here (MPT-VPN); this is a separate instance from the Firefox master, with production-prometheus-vm and bm-xserve05 running a slave for each master
IGNORE THIS (no longer needed now that FF2 is EOL'd): Stop nightly builds on ~~bm-xserve05~~bm-xserve02 (~~~cltbld $ buildbot stop /builds/slave/~~as cltbld, go to ~/cltbld and run "./tinderbox multi stop") and annotate Mozilla1.8 tinderbox tree to explain its absence

Login to productioion-1.8-master as cltbld and start automation run:

buildbot sendchange --username=joduinn --master=localhost:9990 -m"Thunderbird 2.0.0.22 build 1" release

Tag

automated - no problems

Source

automated - no problems

Build/Repack

automated - no problems

Sign

Followed Signing doc
Because of long delay caused by signing of FF3.5beta99 and FF3.5beta99 respin, the "wait for signing log" timed out after 24 hours. Had to:
- Logged into production-1.8-master as buildmaster
- manually hack out tag/srctarball/build steps, then reconfig master
- login to production-1.8-master as cltbld
- start automation run:

buildbot sendchange --username=joduinn --master=localhost:9990 -m"Thunderbird 2.0.0.22 build 1" release

l10n verification

automated - no problems

Generate updates

automated - no problems

Publish updates to test channels (betatest & releasetest)

automated - no problems

Update verify

linux was fine
win32, mac hit problems caused by bootstrap bug 496491. nthomas told me to remove /builds/verify/thunderbird-2.0.0.22/ and retry. This worked a treat.

QA found that sha512 checksums do not work during testing of complete updates. TB2.0.0.22 is the first release using newer sha512 checksums, and this was broken in TB2.0.0.0...TB2.0.0.6. It works fine in TB2.0.0.9 and later. The binaries, and the mar files are all fine. On Monday, we'll regenerate the snippets using old sha1 checksums. For details see bugs attached to tracking bug 496271.

Fix update snippets

# cltbld@production-prometheus-vm
cd /builds/updates/thunderbird-2.0.0.22/patcher
cvs up -r UPDATE_PACKAGING_R7 patcher2.pl MozBuild/Util.pm

cd temp/thunderbird/2.0.0.21-2.0.0.22
mkdir SHA512-snippets
mv aus2* SHA512-snippets

cd /builds/updates/thunderbird-2.0.0.22/patcher
./patcher2.pl --create-patchinfo --app=thunderbird --config=../config/moz18-branch-patcher2.cfg

cd temp/thunderbird/2.0.0.21-2.0.0.22
# did some checks here, diff against SHA512 and grep for hashFunction
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2/ \
 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090614-Thunderbird-2.0.0.22-SHA1
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.beta/ \
 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090614-Thunderbird-2.0.0.22-SHA1-beta
rsync -av -e 'ssh -i /home/cltbld/.ssh/aus' aus2.test/ \
 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090614-Thunderbird-2.0.0.22-SHA1-test

# cltbld@aus2-staging
~/bin/backupsnip 20090614-Thunderbird-2.0.0.22-SHA1-test
~/bin/pushsnip   20090614-Thunderbird-2.0.0.22-SHA1-test

Update verify was green on all three platforms.

Push updates to beta channel

Skipping backupsnip, because it was just done by bhearsum for FF3.5rc1build1.

Waited for formal "go" to beta.

$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ time ~/bin/pushsnip 20090614-Thunderbird-2.0.0.22-SHA1-beta
real    1m42.893s
user    0m0.179s
sys     0m7.826s

Sign installers

Done manually using these installer-signing-instructions here
complete stage-merged:

# on stage
cd /data/cltbld/thunderbird-2.0.0.22/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/thunderbird-2.0.0.22/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:thunderbird .
chmod 644 *SUMS

Update bouncer

Done

Push to mirrors

on stage (as cltbld)

rsync -av /data/cltbld/thunderbird-2.0.0.22/stage-merged/ /home/ftp/pub/thunderbird/releases/2.0.0.22/

Update rsync module

On stage.mozilla.org, add the new release to the smaller, faster, rsync module:

# on stage (as cltbld)
cd /pub/mozilla.org/zz/
vi rsyncd-mozilla-current.exclude

and replace the previous latest release (Tb2.0.0.21) with:

+ thunderbird/releases/2.0.0.22
+ thunderbird/releases/2.0.0.22/**

Final verification

Verify that releasetest points to valid bouncer links:

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz18-thunderbird-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20090610-Thunderbird-2.0.0.22
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090610-Thunderbird-2.0.0.22-test/$a");'

Publish updates to release channel

In case you need to cancel/abort updates, the quickest way to do this is here
While waiting for QA to finish testing, do a backup:

# on aus2-staging verify we have correct tools
$ sudo su - cltbld
$ cd ~/bin
$ cvs update -dP
# backup snippets on release/live channel
$ time ~/bin/backupsnip 20090614-Thunderbird-2.0.0.22-SHA1
real    48m15.344s
user    0m43.540s
sys     1m12.725s

After QA finished testing the releasetest channel and go from release driver, enable release channel:

# on aus2-staging
# put snippets on release/live channel
$ sudo su - cltbld
$ time ~/bin/pushsnip 20090614-Thunderbird-2.0.0.22-SHA1

Update symlink

$ cd /pub/mozilla.org/thunderbird/releases

# confirm where latest symlinks point to where before starting
$ ls -la latest*
lrwxrwxrwx  1 cltbld thunderbird 8 Mar 18 13:54 latest -> 2.0.0.21
lrwxrwxrwx  1 cltbld thunderbird 8 Mar 18 13:54 latest-2.0 -> 2.0.0.21
$ rm latest    # (note no trailing slash)
$ ln -s 2.0.0.22 latest
$ rm latest-2.0   # (note no trailing slash)
$ ln -s 2.0.0.22 latest-2.0
# confirm latest symlinks point to newest release
$ ls -la latest*
lrwxrwxrwx  1 cltbld mozilla 8 Jun 25 17:43 latest -> 2.0.0.22
lrwxrwxrwx  1 cltbld mozilla 8 Jun 25 17:44 latest-2.0 -> 2.0.0.22

Misc

Undid hackery in master.cfg after signing step.

Module	Parent Branch	Created Tag/Branch	Pull date
cvsroot/mozilla	MOZILLA_1_8_BRANCH	GECKO181_20090603_RELBRANCH	2009-06-03 11:32 PDT
cvsroot/mozilla	GECKO181_20090603_RELBRANCH	THUNDERBIRD_2_0_0_22_RELEASE, THUNDERBIRD_2_0_0_22_BUILD1	2009-06-03 11:32 PDT (but see Tag below)
l10n/l10n	MOZILLA_1_8_BRANCH	GECKO181_20090603_RELBRANCH	2009-06-03 11:32 PDT
l10n/l10n	GECKO181_20090603_RELBRANCH	THUNDERBIRD_2_0_0_22_RELEASE, THUNDERBIRD_2_0_0_22_BUILD1	2009-06-03 11:32 PDT

Releases/Thunderbird 2.0.0.22/BuildNotes

Contents

Build Engineers

Bonsai queries

Tags

Build data

Notes

Build 1

Tag

Source

Build/Repack

Sign

l10n verification

Generate updates

Publish updates to test channels (betatest & releasetest)

Update verify

Fix update snippets

Push updates to beta channel

Sign installers

Update bouncer

Push to mirrors

Update rsync module

Final verification

Publish updates to release channel

Update symlink

Misc

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools