Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• bug 754291 - buildbot master Makefile refers to the wrong python
• bug 754325 - release_sanity.py does not recognize Thunderbird configs
• bug 755784 - purge_builds.py reclaims space well beyond what is needed
• 1st signing attempt failed trying to checkout the signing tools from the mofo private CVS repo
  • catlee suggested changing the key used for cvs in ~/.ssh/config. Switched it to the default (id_dsa) and it worked fine.
• bug 756666 - Thunderbird updates builder needs to build patcher tools from mozilla repo, not comm repo
• bug 756599 - thunderbird release 'updates' failed to run backupsnip
• bug 756661 - check_permissions fails because thunderbird release files are not owned by group firefox
• bug 756685 - push_to_mirrors fails for thunderbird when using ffxbld keys

Build Engineers

jhopkins - Tracking bug: bug 744601

Signed-off Revision(s)

Build 1:

• comm-beta: http://hg.mozilla.org/releases/comm-beta/rev/b4e8ce60d23f
• mozilla-beta: http://hg.mozilla.org/releases/mozilla-beta/rev/f40f8e40bae8

L10N changesets

Instructions on how to get them

• Thunderbird: https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=tb13_beta_b2

Tags

Manually tag the automation code, then record the generated tags below. (details)

buildbot-configs:

• THUNDERBIRD_13_0b2_BUILD1 5990:8106e33691c7
• THUNDERBIRD_13_0b2_RELEASE 5988:be09393fd6af

buildbotcustom:

• THUNDERBIRD_13_0b2_RELEASE 2319:62835b2fcaa9
• THUNDERBIRD_13_0b2_BUILD1 2318:e4b03d286da2

tools:

• THUNDERBIRD_13_0b2_RELEASE 2536:e3e998673505
• THUNDERBIRD_13_0b2_BUILD1 2535:5b54bae472ab

Build data

Thunderbird

Notes

Build 1

Diff mozconfigs

If this is a *.0 beta -> release. May also need to update the whitelist of nightly vs release differences for release sanity.

Preparing to start Automation

detailed instructions

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
  • clobberer
  • *NOTE* Did not clobber 13.0b2 build 1 since release-comm-beta wasn't present on the clobberer list yet (first run on FF infra)
• Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:

[cltbld@buildbot-master34 master]$ pwd
/builds/buildbot/build1/master
[cltbld@buildbot-master34 master]$ cat reserved_slaves
8

• For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# ffxbld@stage
cd /pub/mozilla.org/thunderbird/nightly/
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates

• Update l10n changesets for desktop, mobile
  • Instructions on how to get them
  • dashboard
• Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
  • hg tag -f THUNDERBIRD_13_0b2_{RELEASE,BUILD1}
• update and reconfigure the master

cd /builds/buildbot/build1
make update && make checkconfig && make reconfig

• run release sanity

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins -V 13.0b2 \
  --branch mozilla-beta --build-number 1 --release-config release-thunderbird-comm-beta.py \
  --products thunderbird --dryrun localhost:9001

• • release sanity doesn't work properly for Thunderbird yet, so a manual sendchange was issued:

cd /builds/buildbot/build1
source bin/activate
buildbot sendchange --username=jhopkins --master localhost:9001 --branch=releases/comm-beta \
  --comments "THUNDERBIRD 13.0b2 BUILD 1" -p products:thunderbird \
  -p script_repo_revision:THUNDERBIRD_13_0b2_BUILD1 release_build

• start automation ← monitor progress on buildbot (e.g. release-comm-beta)
• reconfigure other masters (or ask the buildduty person to do this) if you have done a merge (and not double-land) to production

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.

Thunderbird,thunderbird,thunderbird,13.0b2,13

E-mail mirrors@mozilla.org

Not necessary for rapid release betas, files only go as far as the 'internal mirrors'.

Edit rsync exclude files

For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.

Tag

No problems.

Bouncer Submitter

• 1st attempt failed due to missing thunderbird-tuxedo.ini config file
• 2nd attemp succeeded

Source

No problems.

Build

Thunderbird

• Linux:
• Linux64:
• Mac:
• Win32:

Thunderbird repacks

• Linux:
• Linux64:
• Mac:
• Win32:

Thunderbird Signing

VERSION=13.0b2
BUILD=1
TAG=THUNDERBIRD_13_0b2_RELEASE
REPO=releases/comm-beta

PRODUCT=thunderbird
APP_NAME=mail
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=fake
STAGE_HOST=stage.mozilla.org
STAGE_USERNAME=tbirdbld
SSH_KEY=$HOME/.ssh/tbirdbld_dsa

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make tb-all PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
    KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \
    APP_NAME=$APP_NAME TAG=$TAG \
    STAGE_USERNAME=$STAGE_USERNAME \
    STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY

• 1st attempt failed trying to checkout the signing tools from the mofo private CVS repo
  • catlee suggested changing the key used for cvs in ~/.ssh/config. Switched it to the default (id_dsa) and it worked fine.
• checked output log

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
    KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \
    APP_NAME=$APP_NAME TAG=$TAG \
    STAGE_USERNAME=$STAGE_USERNAME \
    STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY

Updates

• patcher_build_tools failed
  • filed bug 756666 - Thunderbird updates builder needs to build patcher tools from mozilla repo, not comm repo
  • patched ReleaseUpdatesFactory locally to set mozRepoPath, and reconfiged
  • forced builder again
• backupsnip failed with:

bash: /home/tbirdbld/bin/backupsnip: No such file or directory

• filed bug 756599 - thunderbird release 'updates' failed to run backupsnip
  • To fix, did:

# ffxbld@aus3-staging
chmod 755 /home/ffxbld

• • Then as tbirdbld@aus3-staging

cd /opt/aus2/snippets/staging
~ffxbld/bin/backupsnip Thunderbird-13.0b2-build1-test
~ffxbld/bin/pushsnip Thunderbird-13.0b2-build1-test

• • Then as cltbld@linux64-ix-slave03:

cd /builds/slave/tb-rel-comm-beta-updates/build/temp/thunderbird/13.0b1-13.0b2
rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' aus2/ tbirdbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Thunderbird-13.0b2-build1
bash /builds/slave/tb-rel-comm-beta-updates/tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest

• • Then did the following to get the automation going again:

# cltbld@buildbot-master34
mysql -u buildbot2 -p -h buildbot-rw-vip.db.scl3.mozilla.com buildbot_schedulers
update schedulers set state='{"remainingBuilders": [], "upstreamBuilders": ["release-comm-beta-updates"], "lastReset": 1336754341.071274, "lastCheck": 1336754341.071274}' where name='release-comm-beta-updates_done';
update schedulers set state='{"remainingBuilders": [], "upstreamBuilders": ["release-comm-beta-thunderbird_source", "release-comm-beta-linux_build", "release-comm-beta-linux_repack_complete", "release-comm-beta-linux64_build", "release-comm-beta-linux64_repack_complete", "release-comm-beta-win32_build", "release-comm-beta-win32_repack_complete", "release-comm-beta-macosx64_build", "release-comm-beta-macosx64_repack_complete", "release-comm-beta-updates"], "lastReset": 1336754341.0741279, "lastCheck": 1337318106}' where name='release-comm-beta-thunderbird_deliverables_ready';
update schedulers set state='{"remainingBuilders": ["release-comm-beta-push_to_mirrors"], "upstreamBuilders": ["release-comm-beta-push_to_mirrors", "release-comm-beta-updates"], "lastReset": 1336754341.080065, "lastCheck": 1336754341.080065}' where name='release-comm-beta-thunderbird_uptake_check';

Update verify

• bhearsum fired off the initial round of update_verify a little too soon (by his own admission), so some builders failed to find updates.
• 2nd round was fine

Reset reserved slaves

Check permissions / AV scan

• check permissions failed because it's looking for files owned by group 'firefox'

find /pub/mozilla.org/thunderbird/nightly/13.0b2-candidates/build1/ ! -group firefox ! -path \'*/contrib*\' | wc -l)

• ignored output for now
• filed bug 756661 - check_permissions fails because thunderbird release files are not owned by group firefox

Push to internal mirrors

• automation failed because it was trying to create release dir as ffxbld

ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org mkdir -p /pub/mozilla.org/thunderbird/releases/13.0b2/

• • re-ran the step by hand on mv-moz2-linux-ix-slave09.build.mozilla.org with --ssh-user and --ssh-key set

[cltbld@mv-moz2-linux-ix-slave09 tb-rel-comm-beta-psh-mrrrs]$ scripts/scripts/release/push-to-mirrors.sh mozilla/production_config.py push '--extra-excludes=*.zip' '--extra-excludes=*.zip.asc' --ssh-user tbirdbld --ssh-key tbirdbld_dsa

• • proper solution is to make sure we point to the proper config (i.e. probably not mozilla/production_config.py)
  • filed bug 756685 - push_to_mirrors fails for thunderbird when using ffxbld keys

Final verification

Record receipt of "completed_final_verification" emails (1 per platform)

Going to Beta

Run backupsnip

NOTE: Remember to do this at least 2 hours ahead of the expected "go to beta" email. details

# tbirdbld@aus3-staging.mozilla.org
$ cd /opt/aus2/snippets/staging/
$ time ~/bin/backupsnip Thunderbird-13.0b2-build1

Done in 8 minutes

Push to beta

• Wait for "go" from release managers
• Publish (and record run time here): details

# tbirdbld@aus3-staging.mozilla.org
$ cd /opt/aus2/snippets/staging/
$ time ~/bin/pushsnip Thunderbird-13.0b2-build1

• on completion, email r-d that VERSION updates are live on the beta channel

pushsnip ran in 4m 14s; email sent to r-d

Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.