Until we're about to ship this page will serve as a spot to put down a rough checklist of what needs to be done to ship. Troubleshooting steps and more information here: https://intranet.mozilla.org/MoMo/Build/Releases

Release Engineers

jhopkins

Start the Build

Clobber production release builders at https://clobberer-admin.mozillamessaging.com/ (at the top of the page, choose the production master, comm-release, any builder)
Create release config in the 'release' section of release_master.py
ssh buildbot@momo-vm-03.sj.mozillamessaging.com
cd production/buildbot
hg pull -u
cd /buildbot/production
./buildbot.sh checkconfig
./buildbot.sh reconfig # if the checkconfig passed
Wait 5 minutes
buildbot sendchange --username=standard8 --master=localhost:9010 --branch=releases/comm-release -m "Thunderbird 11 build 2 kick 1" doit
Monitor the build progress at http://build.mozillamessaging.com/buildbot/production/

Sign the Build

VNC to momo-signing01.sj.mozillamessaging.com
Open the SIGNING-README.txt and follow the instructions.

Generate Updates

Once builds are signed and uploaded, updates_release will fire automatically. updates_release update generation has been failing for 11.0 build 1 and build 2 at random points. We've continued update generation manually on the build host, but to hedge our bets we're also generating updates manually on a physical Linux machine. Steps:

First, take linux-ix-slave03 offline
SSH to linux-ix-slave03
cd /builds/slave
mkdir updates_release
cd updates_release
hg clone http://hg.mozilla.org/build/tools tools
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R15 -d build mozilla/tools/patcher
cd build
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R15 -d MozBuild mozilla/tools/release/MozBuild
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R15 -dBootstrap mozilla/tools/release/Bootstrap/Util.pm
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -dpatcher-configs mozilla/tools/patcher-configs
wget -O shipped-locales http://hg.mozilla.org/releases/comm-release/raw-file/THUNDERBIRD_11_0_RELEASE/mail/locales/shipped-locales
wget -O old-shipped-locales http://hg.mozilla.org/releases/comm-release/raw-file/THUNDERBIRD_10_0_2_RELEASE/mail/locales/shipped-locales
# 7 minutes
perl patcher2.pl --build-tools-hg --tools-revision=UPDATE_PACKAGING_R15 --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg
# 23 minutes
perl patcher2.pl --download --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg
cd temp/thunderbird
ln -s 11.0 10.0.2-11.0
cd ../..
# 65 minutes
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg
cd temp/thunderbird/10.0.2-11.0
mv aus2 20110313-Thunderbird-11.0
mv aus2.test 20110313-Thunderbird-11.0-test
tar czf 20110313-Thunderbird-11.0.tgz 20110313-Thunderbird-11.0
tar czf 20110313-Thunderbird-11.0-test.tgz 20110313-Thunderbird-11.0-test
scp *tgz cltbld@momo-build-adm01.sj.mozillamessaging.com:/opt/aus/snippets/staging/
# push snippets to test channels

In the end, the updates we finished generating on the original builder completed, so we used those, not the updates generated on the Firefox build host.

Here are the steps that we ended up using to complete the partially-complete updates_release build:

cd updates_release/build/
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=thunderbird --brand=Thunderbird --config=patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg
cd ..
find build/temp/thunderbird/10.0.2-11.0/ftp/thunderbird/nightly/11.0-candidates/build2 -type f -exec chmod 644 {} ;
find build/temp/thunderbird/10.0.2-11.0/ftp/thunderbird/nightly/11.0-candidates/build2 -type f -exec chmod 644 {} \;
find build/temp/thunderbird/10.0.2-11.0/ftp/thunderbird/nightly/11.0-candidates/build2 -type d -exec chmod 755 {} \;
cd build/temp/thunderbird/10.0.2-11.0/ftp/thunderbird/nightly/11.0-candidates/build2/
ls update/
find . -name '*.mar'
rsync -av -e 'ssh -oIdentityFile=~/.ssh/tbirdbld_dsa' --exclude=*complete.mar update tbirdbld@stage.mozilla.org:/pub/mozilla.org/thunderbird/nightly/11.0-candidates/build2/
cd ../../../../..
less aus2.test/Thunderbird/10.0.2/Linux_x86-gcc3/20120216022139/de/betatest/complete.txt
rsync -av -e 'ssh -oIdentityFile=~/.ssh/tbirdbld_dsa' aus2.test/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/20120313-Thunderbird-11.0-test
bash -c 'ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com "~/bin/backupsnip 20120313-Thunderbird-11.0-test"'
bash -c 'ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa momo-build-adm-01.sj.mozillamessaging.com "~/bin/pushsnip 20120313-Thunderbird-11.0-test"'
rsync -av -e 'ssh -oIdentityFile=~/.ssh/tbirdbld_dsa' aus2/ tbirdbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus2/snippets/staging/20120313-Thunderbird-11.0
bash /buildbot//updates_release/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

Major Updates

edit patcher-configs

# 3.1.x->release
$ cd ~/cvs/mozilla/tools/patcher-configs
$ cvs update
$ vi moz192-thunderbird-branch-patcher2.cfg
- yank 3.1.20
- e moz20-thunderbird-branch-major-update-patcher2.cfg
- paste
- remove older releases from the bottom of the file
- e mozRelease-thunderbird-branch-patcher2.cfg
- yank 11.0
- e moz20-thunderbird-branch-major-update-patcher2.cfg
- paste
- edit lines between <current-update>
  - version, build number
  - from, to
- attach the patch to your major updates bug and get review
- commit your changes

Generate major updates

First, take linux-ix-slave03 offline
SSH to linux-ix-slave03
cd /builds/slave
mkdir major_updates_31
cd major_updates_31
hg clone http://hg.mozilla.org/build/tools tools
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R11_1_MU -d build mozilla/tools/patcher
cd build
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R11_1_MU -d MozBuild mozilla/tools/release/MozBuild
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R11_1_MU -dBootstrap mozilla/tools/release/Bootstrap/Util.pm
cvs -d :ext:tbirdbld@cvs.mozilla.org:/cvsroot co -dpatcher-configs mozilla/tools/patcher-configs

wget -O shipped-locales http://hg.mozilla.org/releases/comm-release/raw-file/THUNDERBIRD_11_0_RELEASE/mail/locales/shipped-locales
wget -O old-shipped-locales http://hg.mozilla.org/releases/comm-1.9.2/raw-file/THUNDERBIRD_3_1_20_RELEASE/mail/locales/shipped-locales

# 10 minutes
perl patcher2.pl --build-tools-hg --tools-revision=UPDATE_PACKAGING_R11_1_MU --app=thunderbird --brand=Thunderbird --config=patcher-configs/moz20-thunderbird-branch-major-update-patcher2.cfg
# 23 minutes
perl patcher2.pl --download --app=thunderbird --brand=Thunderbird --config=patcher-configs/moz20-thunderbird-branch-major-update-patcher2.cfg
cd temp/thunderbird
ln -s 11.0 3.1.20-11.0
cd ../..
# 65 minutes
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=thunderbird --brand=Thunderbird --config=patcher-configs/moz20-thunderbird-branch-major-update-patcher2.cfg
cd temp/thunderbird/3.1.20-11.0
mv aus2 20110314-Thunderbird-3.1.20-11.0-MU
mv aus2.test 20110314-Thunderbird-3.1.20-11.0-MU-test
mv aus2.beta 20110314-Thunderbird-3.1.20-11.0-MU-beta
tar czf 20110314-Thunderbird-3.1.20-11.0-MU.tgz 20110314-Thunderbird-3.1.20-11.0-MU
tar czf 20110314-Thunderbird-3.1.20-11.0-MU-test.tgz 20110314-Thunderbird-3.1.20-11.0-MU-test
tar czf 20110314-Thunderbird-3.1.20-11.0-MU-beta.tgz 20110314-Thunderbird-3.1.20-11.0-MU-beta
# in an empty directory on your workstation
scp cltbld@linux-ix-slave03.build.scl1.mozilla.com:/builds/slave/major_updates_31/build/temp/thunderbird/3.1.20-11.0/*.tgz .
scp *.tgz cltbld@momo-build-adm-01.sj.mozillamessaging.com:/opt/aus/snippets/staging/
ssh cltbld@momo-build-adm-01.sj.mozillamessaging.com
cd /opt/aus/snippets/staging
ls -ltr
tar zxf 20110314-Thunderbird-3.1.20-11.0-MU-beta.tgz
tar zxf 20110314-Thunderbird-3.1.20-11.0-MU.tgz
tar zxf 20110314-Thunderbird-3.1.20-11.0-MU-test.tgz
pushsnip 20110314-Thunderbird-3.1.20-11.0-MU-test
# then announce to #tbdrivers and thunderbird-drivers@mozilla.org

NOTE: alternatively, we could have used buildbot to generate major updates:

$ vi buildbot-configs/thunderbird/release_master.py
  - example: https://hg.mozilla.org/build/buildbot-configs/rev/9e1a085c6b9d
- pull/reconfigure buildbot
- kick off the major_update_31 builder

But the update generation on our VMs has been finicky lately (failing midway) and takes much longer than a physical host.

Send a notification email

Let people know what's been done. Note: this is a wide list so keep traffic low.

To: thunderbird-drivers@mozilla.org
Subj: Thunderbird 11.0 is available on test channels <eom>"

Redirects

Use the svn:sys/scripts/insert-website-redirect.pl to insert redirects into puppet/modules/apache/files/conf.d/live.mozillamessaging.com.conf