Until we're about to ship this page will serve as a spot to put down a rough checklist of what needs to be done to ship.

General Notes

See Thunderbird release docs

Release Engineers

rail, tracking bug: bug 744150

Bugs Hit

there was no 12.0 milestone, had to use 12.0b4 milestone
Updates failed
bug 747689 - Thunderbird update verify fails trying to download 10.0

Build Data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120420153403	momo-vm-linux-03	1 hrs, 43 mins, 28 secs
	Linux64	20120420153206	momo-vm-linux64-06	1 hrs, 21 mins, 5 secs
	Mac	20120420154114	tb2-darwin10-slave62	3 hrs, 37 mins, 27 secs
	Windows	20120420153905	momo-vm-win2k3-08	1 hrs, 52 mins, 45 secs

Build 1

Signed off revisions

comm-release: http://hg.mozilla.org/releases/comm-release/rev/37620e34f3a4
mozilla-release: Same changeset as for Firefox 12.0 final. Let buildbot create its own relbranch. (5bcfa0da3be9)
L10n-changesets: sign-off 12.0 on the dashboard (I think there's one minor zh-TW change, ignore any pending). Took 12.0b4 changeset.

Start the Build

Preparation

Clobberred comm-beta branch at https://clobberer-admin.mozillamessaging.com/ (at the top of the page, choose the production master, comm-release, any builder)
Create release config patch
Add L10N changesets
Land configs
Tag configs and tools using

hg tag THUNDERBIRD_12_0_RELEASE THUNDERBIRD_12_0_BUILD1

Update and reconfig the master

ssh buildbot@momo-vm-03.sj.mozillamessaging.com
cd ~/production/buildbot-configs
hg pull -u
# check if there are local chanes
hg diff
cd /buildbot/production
./buildbot.sh checkconfig
./buildbot.sh reconfig & tail -F buildbot-configs/thunderbird/twistd.log |grep configuration

Start the automation

buildbot sendchange --username=rail --master=localhost:9010 --branch=releases/comm-release -m "Thunderbird 12.0 build 1 kick 1" doit

Monitor the build progress at http://build.mozillamessaging.com/buildbot/production/ and manage it at https://buildbot-admin.mozillamessaging.com/production/

Tag

No problems

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org"

Subject: Thunderbird 12.0

Thunderbird,thunderbird,thunderbird,12.0,12.0

E-mail mirrors@mozilla.org

Not necessary for rapid release betas or ESR builds; files only go as far as the 'internal mirrors'.

Coordinate yourself with the person creating the ESR and the latest release which normally go out at the same time.

They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time.

Subject: Thunderbird 12.0 coming to mirrors on Apr 24, 2012
Body:
We're planning on pushing our Thunderbird 12.0 release to mirrors ~24 hours before Tuesday, April 24, 2012.
# If this is a chemspill, please mention that in the email as it will mean pinging 
# for CDN to be enabled when the push to internal mirrors happens

If you believe these releases or these dates will cause any problems,
please notify release at mozilla.com. If you have any problems with mirror
status/weights/etc, please notify mirror-submissions at mozilla.org.

Source

No problems.

Builds

No problems

L10N repacks

linux sanity check:

 check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164  --max 217  linux

linux64 sanity check:

 check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164  --max 217  linux64

win32 sanity check:

 check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 180  --max 233  win32

mac sanity check

check-l10n-release-builds.pl --repo http://hg.mozilla.org/releases/comm-release --branch=release --revision THUNDERBIRD_12_0_RELEASE --min 164  --max 217   osx

Signing

Using Firefox signing machine

VERSION=12.0
BUILD=1
TAG=THUNDERBIRD_12_0_RELEASE
REPO=releases/comm-release

PRODUCT=thunderbird
APP_NAME=mail
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=fake
STAGE_HOST=stage.mozilla.org
STAGE_USERNAME=tbirdbld
CVS_USER=tbirdbld
SSH_KEY=$HOME/.ssh/tbirdbld_dsa

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make tb-all PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
    KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \
    APP_NAME=$APP_NAME TAG=$TAG \
    STAGE_USERNAME=$STAGE_USERNAME \
    STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY \
    CVS_USER=$CVS_USER

run the same with "postsign" target

L10N Verify

Updates

First run failed

Adding New File Contents/MacOS/components/libalerts.dylib
done with patch 119/220 time (79.12s/16.19s/95.31s) (mar/patch/total)
Extracting /buildbot/updates_release/build/temp/thunderbird/11.0.1/ftp/thunderbird-11.0.1.de.mac.complete.mar to /buildbot/updates_release/build/tmpgQPar3-fastmode/120/from
Extracting /buildbot/updates_release/build/temp/thunderbird/12.0/ftp/thunderbird-12.0.de.mac.complete.mar to /buildbot/updates_release/build/tmpgQPar3-fastmode/120/to

IN SHUTDOWN...
Failed to remove patcher2.pid: No such file or directory
FAIL shell call timed out after 10800 seconds at patcher2.pl line 1682.
program finished with exit code 2
elapsedTime=10846.638534

Commented out the following steps

       #self.bumpPatcherConfig()
       #self.bumpVerifyConfigs()

reconfigured and forced the updates builder
second run failed as well
Ran it manually on the same slave (no need to build update tools)

# cltbld@momo-vm-linux-03.sj.mozillamessaging.com

cd /buildbot/updates_release/build
rm -rf temp
perl patcher2.pl --download --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg
export MAR=/buildbot/updates_release/build/mozilla/obj/dist/host/bin/mar
find . -name '*.mar' | while read marname; do $MAR -t $marname > /dev/null; x=$?; if [ $x != "0" ]; then echo "FAIL $marname"; fi; done
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg

find temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1 -type f -exec chmod 644 {} \;
find temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1 -type d -exec chmod 755 {} \;

cd temp/thunderbird/11.0.1-12.0/ftp/thunderbird/nightly/12.0-candidates/build1
rsync -avP -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" --exclude=*complete.mar update tbirdbld@stage.mozilla.org:/pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/

cd -
cd temp/thunderbird/11.0.1-12.0/

rsync -av -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" aus2.test/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/Thunderbird-12.0-build1-test
rsync -av -e "ssh -oIdentityFile=~/.ssh/tbirdbld_dsa" aus2/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/Thunderbird-12.0-build1
bash /buildbot/updates_release/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com "~/bin/pushsnip Thunderbird-12.0-build1-test"

Update Verify

Forced update verifies failed trying to download 10.0 from a usual location, while the directory should be 10.0-real. Filed bug 747689
No problems after landing bug 747689

Send a notification email

Let people know what's been done. Note: this is a wide list so keep traffic low.

To: thunderbird-drivers@mozilla.org
Subj: Thunderbird 12.0 build1 is available on test channels

Major Updates

Configs landed (as a part of 12.0 patch)
Set clobberer
forced the builder
first build timed out trying to remove old builds
forced again
Second run failed with

wget of http://stage.mozilla.org//pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/macosx_info.txt failed. at MozBuild/Util.pm line 603.
program finished with exit code 1

Set clobberer
Copied macosx_info.txt:

# tbirdbld@surf
cd /pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1/
cp macosx64_info.txt macosx_info.txt

Forced 3rd builder

Major Update Verify

Add to Bouncer

pushed a braindump patch
Ran the following

# from my laptoop, in tools/release
$ bash ../../braindump/releases-related/add_thunderbird_to_bouncer.sh 
Adding product: Thunderbird-12.0
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1673" name="Thunderbird-12.0"/></products>
Adding product: Thunderbird-12.0-Complete
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1674" name="Thunderbird-12.0-Complete"/></products>
Adding product: Thunderbird-12.0-Partial-11.0.1
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1675" name="Thunderbird-12.0-Partial-11.0.1"/></products>
Adding location for Thunderbird-12.0, linux: /thunderbird/releases/12.0/linux-i686/:lang/thunderbird-12.0.tar.bz2
Adding location for Thunderbird-12.0-Complete, linux: /thunderbird/releases/12.0/update/linux-i686/:lang/thunderbird-12.0.complete.mar
Adding location for Thunderbird-12.0-Partial-11.0.1, linux: /thunderbird/releases/12.0/update/linux-i686/:lang/thunderbird-11.0.1-12.0.partial.mar
Adding location for Thunderbird-12.0, linux64: /thunderbird/releases/12.0/linux-x86_64/:lang/thunderbird-12.0.tar.bz2
Adding location for Thunderbird-12.0-Complete, linux64: /thunderbird/releases/12.0/update/linux-x86_64/:lang/thunderbird-12.0.complete.mar
Adding location for Thunderbird-12.0-Partial-11.0.1, linux64: /thunderbird/releases/12.0/update/linux-x86_64/:lang/thunderbird-11.0.1-12.0.partial.mar
Adding location for Thunderbird-12.0, osx: /thunderbird/releases/12.0/mac/:lang/Thunderbird%2012.0.dmg
Adding location for Thunderbird-12.0-Complete, osx: /thunderbird/releases/12.0/update/mac/:lang/thunderbird-12.0.complete.mar
Adding location for Thunderbird-12.0-Partial-11.0.1, osx: /thunderbird/releases/12.0/update/mac/:lang/thunderbird-11.0.1-12.0.partial.mar
Adding location for Thunderbird-12.0, win: /thunderbird/releases/12.0/win32/:lang/Thunderbird%20Setup%2012.0.exe
Adding location for Thunderbird-12.0-Complete, win: /thunderbird/releases/12.0/update/win32/:lang/thunderbird-12.0.complete.mar
Adding location for Thunderbird-12.0-Partial-11.0.1, win: /thunderbird/releases/12.0/update/win32/:lang/thunderbird-11.0.1-12.0.partial.mar

Push to the internal mirrors

Added the following entry to /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

- thunderbird/releases/12.0

# ~/bin/push-release /pub/mozilla.org/thunderbird/nightly/12.0-candidates/build1 /pub/mozilla.org/thunderbird/releases/12.0

Copy/paste the output and run it (took ~10mins)

Final Verification