Checklist

~~Set reserved_slaves ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Check to make sure you have all of the different types of slaves that you need ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Start the automation ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Reset reserved_slaves and unlock slaves ^DESKTOP ^MOBILE ^RELEASE ^BETA ^ESR~~
~~Push to mirrors ^DESKTOP ^RELEASE~~
~~Run pushsnip ^DESKTOP ^RELEASE ^BETA ^ESR~~

Bugs hit

bug 799806
- need extra mozconfig line --enable-stdcxx-compat to build libmar. bhearsum fixed
- only fixed Firefox the first time round, was a new kind of broken on Thunderbird. nthomas fixed
bug 799347 - Investigate packet loss within SCL3
- network issues caused repacks to fail
- Note for post-mortem Ask relman to advise IT of chemspills.
bug 800277 - Update generation has problems when latest-1 --> latest partial isn't generated
bug 800338 - win32 'id' locale for thunderbird 16.0.1 has wrong checksums filesizes

Notes

Tracking bug bug 800023
Release engineer bhearsum, kmoir, nthomas
Buildbot master: bm34

Build 1

Updated de revision to that used for 16.0 respin
Set reserved slaves to 2
Moved 2 64-bit linux ix slaves, 1 64-bit windows slave, and 2 32-bit windows slaves to bm34 to make sure we have enough of the right slaves to do the release.
Tagged buildbot-configs, buildbotcustom, and tools with THUNDERBIRD_16_0_1_{RELEASE,BUILD1}
Updated and reconfiged bm34
Ran release sanity in dry run mode:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
 -V 16.0.1 --build-number 1 --branch comm-release \
 --release-config release-thunderbird-comm-release.py --product thunderbird \
 --l10n-dashboard-version=16 --dryrun localhost:9001

2012-10-10 14:51:21,104 : INFO : Comparing l10n changesets on dashboard https://l10n.mozilla.org/shipping/l10n-changesets?ms=tb16 to on-disk l10n-changesets_thunderbird-release ...
2012-10-10 14:51:21,151 : ERROR : 	locale de revisions not matching: 435ffc2817d3 (config) vs. 5eb2fb1a7155 (dashboard)
2012-10-10 14:51:21,151 : ERROR : Error verifying l10n dashboard changesets

so the dashboard hasn't got the de fix.

Ran without --dryrun, and with -l to skip l10n check, to start the release

Fix repacks

Due to bug 799806, we need to add --enable-stdcxx-compat to our Linux l10n mozconfigs. This was done in https://hg.mozilla.org/build/buildbot-configs/rev/0029772efc8d. That revision was retagged, and 'rebuild' was used on all of the failed repacks (linux + linux64) to get them going again.

Also needed to handle tier_base and tier_nspr properly for thunberbird, bug 799806 again.

Repacks failed due to network issues in bug 799347 (scl3 core router problem). After enough retries and IT work around they succeeded.

Updates

Update generation builder went red because we don't really support not generating a partial from the previous release, bug 800277.

Generate 16.0 -> 16.0.1 complete update

The locales list didn't change between 15.0.1 and 16.0 so we can just grab the 15.0.1 snippets, munge the buildID and zap the partial.txt files.

 # tbirdbld@aus3-staging
 cd /opt/aus2/snippets/staging/
 mkdir -p Thunderbird-16.0.1-build1-16.0-test/Thunderbird
 rsync -av ../pushed/Thunderbird-16.0.1-build1-test/Thunderbird/15.0.1/ \
          Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/
 for d in Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/*; do 
   mv -v $d/20120907140327 $d/20121005141928
 done
 find Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/ -type f -name partial.txt -exec rm {} \;
 pushsnip Thunderbird-16.0.1-build1-16.0-test

Release snippets, from the same working location

 rsync -av Thunderbird-16.0.1-build1/Thunderbird/15.0.1/ \
           Thunderbird-16.0.1-build1/Thunderbird/16.0/
 for d in Thunderbird-16.0.1-build1/Thunderbird/16.0/*; do 
   mv -v $d/20120907140327 $d/20121005141928
 done
 find Thunderbird-16.0.1-build1/Thunderbird/16.0/ -type f -name partial.txt -exec rm {} \;

Generate update verify configs and trigger

On my own machines

mkdir tmp; cd tmp
hg clone http://hg.mozilla.org/build/tools
wget -O shipped-locales http://hg.mozilla.org/releases/comm-release/raw-file/THUNDERBIRD_16_0_1_RELEASE/mail/locales/shipped-locales

Apply this patch

diff --git a/lib/python/release/updates/patcher.py b/lib/python/release/updates/patcher.py
--- a/lib/python/release/updates/patcher.py
+++ b/lib/python/release/updates/patcher.py
@@ -47,5 +47,5 @@ class PatcherConfig(dict):
         # assume that all of the fromVersions in the past-update lines are
         # versions that should have update paths to the latest on all channels.
-        return tuple([self['current-update']['from']] + [v[0] for v in self['past-update']])
+        return tuple(set([self['current-update']['from']] + [v[0] for v in self['past-update']]))
 
     def getOptionalAttrs(self, version):

to resolve the locale error. Then generate the configs with

python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozRelease-thunderbird-mac64.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozRelease-thunderbird-win32.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozRelease-thunderbird-linux64.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform linux --output tools/release/updates/mozRelease-thunderbird-linux.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE

Then add the 16.0 lines we want by doing this in each config file

copy the two 15.0 lines up to the top of the file, and make some modifications ...
change to release="16.0"
change the build_id to 20121005141928
change the 15.0's to 16.0's in the from="/thunderbird..." vars
move aus_server and to definitions to the first line

Commit and move THUNDERBIRD_16_0_1_{RELEASE,BUILD1}_RUNTIME tags.

Then forced update verify builders (from local machine)

for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4; do
    curl "http://buildbot-master34.build.mozilla.org:8001/builders/release-comm-release-${p}_update_verify_${i}%2F4/force" > /dev/null
    sleep 5
  done
done

Issues with id locale on win32

eg

FAIL: complete from https://aus3.mozilla.org/update/1/Thunderbird/16.0/20121005141928/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size
FAIL: update.xml size: 22681695
FAIL: actual size: 22681738
complete from https://aus3.mozilla.org/update/1/Thunderbird/15.0.1/20120907140327/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size
update.xml size: 22681695
actual size: 22681738
download_mars returned non-zero exit code: 1
partial from https://aus3.mozilla.org/update/1/Thunderbird/15.0.1/20120907140327/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size
update.xml size: 7565330
actual size: 7565337
download_mars returned non-zero exit code: 1

bhearsum did some investigating and couldn't find any obvious cause. bug 800338 was filed to track it down. In the meantime, did the following to fix us up by hand:

# tbirdbld@aus3-staging
cd /opt/aus2/snippets/staging
rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/ Thunderbird-16.0.1-build1-test-fixed-id/
# values on the left side of the replacements are from the checksums file on the slave; values on the right are from the checksums on ftp
find Thunderbird-16.0.1-build1-test-fixed-id -type f -regex '.*/id/.*' -exec sed -i -e 's/22681695/22681738/' -e 's/43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1/fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664/' -e 's/7565330/7565337/' -e 's/3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78/9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e/' {} \;
rsync -av Thunderbird-16.0.1-build1/ Thunderbird-16.0.1-build1.bak/
find Thunderbird-16.0.1-build1 -type f -regex '.*/id/.*' -exec sed -i -e 's/22681695/22681738/' -e 's/43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1/fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664/' -e 's/7565330/7565337/' -e 's/3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78/9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e/' {} \;

Output from some quick verifications:

diff -Naur /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/Thunderbird/ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/ | grep '^++'
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/betatest/complete.txt	2012-10-11 06:19:42.652817000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/releasetest/complete.txt	2012-10-11 06:19:42.656822000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/betatest/complete.txt	2012-10-11 06:19:41.151731000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/releasetest/complete.txt	2012-10-11 06:19:41.155731000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/betatest/complete.txt	2012-10-11 06:19:41.766769000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/releasetest/complete.txt	2012-10-11 06:19:41.771766000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/betatest/complete.txt	2012-10-11 06:19:44.587936000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/releasetest/complete.txt	2012-10-11 06:19:44.591934000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/betatest/complete.txt	2012-10-11 06:19:43.554883000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/releasetest/complete.txt	2012-10-11 06:19:43.559878000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/betatest/complete.txt	2012-10-11 06:19:46.335039000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/releasetest/complete.txt	2012-10-11 06:19:46.339037000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/betatest/complete.txt	2012-10-11 06:19:45.443986000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/releasetest/complete.txt	2012-10-11 06:19:45.448984000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/betatest/complete.txt	2012-10-11 06:19:48.457166000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/releasetest/complete.txt	2012-10-11 06:19:48.462172000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/betatest/complete.txt	2012-10-11 06:19:47.281099000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/releasetest/complete.txt	2012-10-11 06:19:47.286093000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/betatest/complete.txt	2012-10-11 06:19:49.827247000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/releasetest/complete.txt	2012-10-11 06:19:49.832249000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/betatest/complete.txt	2012-10-11 06:19:52.509405000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/releasetest/complete.txt	2012-10-11 06:19:52.514404000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/betatest/complete.txt	2012-10-11 06:19:51.195324000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/betatest/partial.txt	2012-10-11 06:19:51.198326000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/releasetest/complete.txt	2012-10-11 06:19:51.202332000 -0700
+++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/releasetest/partial.txt	2012-10-11 06:19:51.205331000 -0700
diff -Naur /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1.bak/Thunderbird/ Thunderbird-16.0.1-build1/Thunderbird/ | grep '^++'
+++ Thunderbird-16.0.1-build1/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/release/complete.txt	2012-10-11 06:29:26.041502000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/release/complete.txt	2012-10-11 06:29:24.568416000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/release/complete.txt	2012-10-11 06:29:25.353462000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/release/complete.txt	2012-10-11 06:29:27.495594000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/release/complete.txt	2012-10-11 06:29:26.740552000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/release/complete.txt	2012-10-11 06:29:28.945675000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/release/complete.txt	2012-10-11 06:29:28.188630000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/release/complete.txt	2012-10-11 06:29:30.463765000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/release/complete.txt	2012-10-11 06:29:29.664720000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/release/complete.txt	2012-10-11 06:29:31.308823000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/release/complete.txt	2012-10-11 06:29:32.615892000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/release/complete.txt	2012-10-11 06:29:32.012863000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/release/partial.txt	2012-10-11 06:29:32.016859000 -0700
+++ Thunderbird-16.0.1-build1/Thunderbird/16.0/WINNT_x86-msvc/20121005141928/id/release/complete.txt	2012-10-11 06:29:33.429941000 -0700

diff -Naur /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/Thunderbird/ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/ | grep '^[-+]' | grep -v '+++' | grep -v '\-\-\-'
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78
-size=7565330
+hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e
+size=7565337
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78
-size=7565330
+hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e
+size=7565337
diff -Naur /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1.bak/Thunderbird/ Thunderbird-16.0.1-build1/Thunderbird/ | grep '^[-+]' | grep -v '+++' | grep -v '\-\-\-'
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738
-hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78
-size=7565330
+hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e
+size=7565337
-hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1
-size=22681695
+hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664
+size=22681738

Pushed the updated test snippets:

~/bin/pushsnip Thunderbird-16.0.1-build1-test-fixed-id

And retriggered win32 update verify through the "rebuild" button.

The first update verify chunk failed again, but only for the "16.0" update. This is because I forgot to take into account the manually generated 16.0 updates. To fix:

# tbirdbld@aus3-staging
cd /opt/aus2/snippets/staging/
mkdir -p Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird
rsync -av --include=WINNT_x86-msvc --include=20120907140327 --include=id --include=betatest --include=releasetest --include=*.txt \
       --exclude=* ../pushed/Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/ \
         Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/
mv -v Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/WINNT_x86-msvc/20120907140327 Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/WINNT_x86-msvc/20121005141928
find Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/ -type f -name partial.txt -exec rm -v {} \;
~/bin/pushsnip Thunderbird-16.0.1-build1-16.0-test-fixed-id

Manually triggered update verify chunk #1 again.

No need to adjust the release snippets, because they already had 16.0 snippets in them. To be paranoid, used the snippet compare script to verify that releasetest and release matched:

# tbirdbld@aus3-staging
cd ~/
# "repush" snippets into a non-live directory so we have something to compare against.
rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/ Thunderbird-16.0.1-build1-test-complete/
rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-16.0-test/ Thunderbird-16.0.1-build1-test-complete/
rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test-fixed-id/ Thunderbird-16.0.1-build1-test-complete/
rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-16.0-test-fixed-id/ Thunderbird-16.0.1-build1-test-complete/
hg clone http://hg.mozilla.org/build/tools
wget -O- https://hg.mozilla.org/build/tools/archive/tip.tar.gz | gunzip | tar -xv
cd tools-eb240f672142/release
bash compare-channel-snippets.sh /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1 release ~/Thunderbird-16.0.1-build1-test-complete/ releasetest

...and then retriggered ALL update verify builders because I'm super paranoid:

for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4; do
    curl "http://buildbot-master34.build.mozilla.org:8001/builders/release-comm-release-${p}_update_verify_${i}%2F4/force" > /dev/null
    sleep 5
  done
done

Also had to update the checksums files. To do this, manually edited /pub/mozilla.org/thunderbird/nightly/16.0.1-candidates/build1/win32/id/thunderbird-16.0.1.checksums on upload1 to have correct values. With that fixed, used "force build" to start the "checksums" builder again to regenerate the big *SUMS files + signatures. The builder burned when trying to create contrib dirs again, but it had already uploaded the new *SUMS files + sigs, so that's okay. Ran the following to sync the new files to releases/

# tbirdbld@upload1
rsync  -av --include=*SUMS* --exclude=* /pub/mozilla.org/thunderbird/nightly/16.0.1-candidates/build1/ /pub/mozilla.org/thunderbird/releases/16.0.1/

push to mirrors

initiated the following by forcing build on bm34
  release-comm-release-push_to_mirrors
  start update monitoring
  release-comm-release-macosx64_final_verification 
  release-comm-release-win32_final_verification 
  release-comm-release-linux64_final_verification
  release-comm-release-linux_final_verification
  ready for release
  ready for releasetest

pushsnip

 [tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging
 [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-16.0.1-build1

Unthrottle

Standard8 asked us to unthrottle right after pushing. this patch was landed and retagged with AUS2_PRODUCTION. The push was done in bug 800473

Manually stop TriggerBouncerCheck

For some reason, probably related to Bouncer being broken and then fixed (bug 800457, this master kept firing the post-"ready for rel test" builders over and over. To fix, manhole'd in and ran the following:

master.scheduler_manager.namedServices["release-comm-release-ready-for-release"].stopLoop()
master.scheduler_manager.namedServices["release-comm-release-ready-for-rel-test"].stopLoop('shut up already!')

This didn't work. Rail started poking at it afterwards.

A brute force attack via Manhole:

>>> print "\n".join(["%s %s" % (i, str(x)) for i, x in enumerate(master.allSchedulers()) if "TriggerBouncerCheck" in str(x)])
13 <buildbotcustom.scheduler.TriggerBouncerCheck instance at 0x2aaad8dc0a70>
27 <buildbotcustom.scheduler.TriggerBouncerCheck instance at 0x2aaad4249b00>
>>> master.allSchedulers()[13].loop = None
>>> master.allSchedulers()[27].loop = None

I also set the master for a graceful shutdown to cleanup the brute force hacks.

Releases/Thunderbird 16.0.1/BuildNotes

Contents

Checklist

Bugs hit

Notes

Build 1

Fix repacks

Updates

Generate 16.0 -> 16.0.1 complete update

Generate update verify configs and trigger

Issues with id locale on win32

push to mirrors

pushsnip

Unthrottle

Manually stop TriggerBouncerCheck

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools