Releases/Thunderbird 13.0b2/BuildNotes
Contents
- 1 Notes About Releasing
- 2 Bugs hit
- 3 Build Engineers
- 4 Signed-off Revision(s)
- 5 L10N changesets
- 6 Tags
- 7 Build data
- 8 Notes
- 8.1 Build 1
- 8.1.1 Diff mozconfigs
- 8.1.2 Preparing to start Automation
- 8.1.3 E-mail Metrics
- 8.1.4 E-mail mirrors@mozilla.org
- 8.1.5 Edit rsync exclude files
- 8.1.6 Tag
- 8.1.7 Bouncer Submitter
- 8.1.8 Source
- 8.1.9 Build
- 8.1.10 Updates
- 8.1.11 Reset reserved slaves
- 8.1.12 Check permissions / AV scan
- 8.1.13 Push to internal mirrors
- 8.1.14 Final verification
- 8.1.15 Going to Beta
- 8.1 Build 1
Notes About Releasing
Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed
Bugs hit
Enter any bugs pre-existing or newly discovered and filed during the release:
- bug 754291 - buildbot master Makefile refers to the wrong python
- bug 754325 - release_sanity.py does not recognize Thunderbird configs
- bug 755784 - purge_builds.py reclaims space well beyond what is needed
- 1st signing attempt failed trying to checkout the signing tools from the mofo private CVS repo
- catlee suggested changing the key used for cvs in ~/.ssh/config. Switched it to the default (id_dsa) and it worked fine.
- bug 756666 - Thunderbird updates builder needs to build patcher tools from mozilla repo, not comm repo
- bug 756599 - thunderbird release 'updates' failed to run backupsnip
- bug 756661 - check_permissions fails because thunderbird release files are not owned by group firefox
- bug 756685 - push_to_mirrors fails for thunderbird when using ffxbld keys
Build Engineers
jhopkins - Tracking bug: bug 744601
Signed-off Revision(s)
Build 1:
- comm-beta: http://hg.mozilla.org/releases/comm-beta/rev/b4e8ce60d23f
- mozilla-beta: http://hg.mozilla.org/releases/mozilla-beta/rev/f40f8e40bae8
L10N changesets
Instructions on how to get them
Tags
Manually tag the automation code, then record the generated tags below. (details)
buildbot-configs:
- THUNDERBIRD_13_0b2_BUILD1 5990:8106e33691c7
- THUNDERBIRD_13_0b2_RELEASE 5988:be09393fd6af
buildbotcustom:
- THUNDERBIRD_13_0b2_RELEASE 2319:62835b2fcaa9
- THUNDERBIRD_13_0b2_BUILD1 2318:e4b03d286da2
tools:
- THUNDERBIRD_13_0b2_RELEASE 2536:e3e998673505
- THUNDERBIRD_13_0b2_BUILD1 2535:5b54bae472ab
Build data
Thunderbird
Build # | Type | Build ID | Build machine | Time to build |
1 | Linux | 20120512204115 | mv-moz2-linux-ix-slave09 | 36 mins, 8 secs |
Linux64 | 20120512223958 | linux64-ix-slave05 | 28 mins, 52 secs | |
Mac | 20120512204231 | moz2-darwin10-slave40 | 3 hrs, 35 mins, 40 secs | |
Windows | 20120512205549 | mw32-ix-slave07 | 1 hrs, 42 mins, 11 secs |
Notes
Build 1
Diff mozconfigs
If this is a *.0 beta -> release. May also need to update the whitelist of nightly vs release differences for release sanity.
Preparing to start Automation
- Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
- clobberer
- *NOTE* Did not clobber 13.0b2 build 1 since release-comm-beta wasn't present on the clobberer list yet (first run on FF infra)
- Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:
[cltbld@buildbot-master34 master]$ pwd /builds/buildbot/build1/master [cltbld@buildbot-master34 master]$ cat reserved_slaves 8
- For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)
# ffxbld@stage cd /pub/mozilla.org/thunderbird/nightly/ mkdir ../candidates/$version-candidates ln -s ../candidates/$version-candidates $version-candidates
- Update l10n changesets for desktop, mobile
- Instructions on how to get them
- dashboard
- Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
- Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
- hg tag -f THUNDERBIRD_13_0b2_{RELEASE,BUILD1}
- update and reconfigure the master
cd /builds/buildbot/build1 make update && make checkconfig && make reconfig
-
run release sanity
cd /builds/buildbot/build1/master source ../bin/activate PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u jhopkins -V 13.0b2 \ --branch mozilla-beta --build-number 1 --release-config release-thunderbird-comm-beta.py \ --products thunderbird --dryrun localhost:9001
- the same command as above but without the "--dryrun"
- release sanity doesn't work properly for Thunderbird yet, so a manual sendchange was issued:
cd /builds/buildbot/build1 source bin/activate buildbot sendchange --username=jhopkins --master localhost:9001 --branch=releases/comm-beta \ --comments "THUNDERBIRD 13.0b2 BUILD 1" -p products:thunderbird \ -p script_repo_revision:THUNDERBIRD_13_0b2_BUILD1 release_build
- start automation ← monitor progress on buildbot (e.g. release-comm-beta)
- reconfigure other masters (or ask the buildduty person to do this) if you have done a merge (and not double-land) to production
E-mail Metrics
Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.
Thunderbird,thunderbird,thunderbird,13.0b2,13
E-mail mirrors@mozilla.org
Not necessary for rapid release betas, files only go as far as the 'internal mirrors'.
Edit rsync exclude files
For rapid release betas, this is not necessary as there's already an exclude that matches beta releases.
Tag
No problems.
Bouncer Submitter
- 1st attempt failed due to missing thunderbird-tuxedo.ini config file
- 2nd attemp succeeded
Source
No problems.
Build
Thunderbird
- Linux:
- Linux64:
- Mac:
- Win32:
Thunderbird repacks
- Linux:
- Linux64:
- Mac:
- Win32:
Thunderbird Signing
VERSION=13.0b2 BUILD=1 TAG=THUNDERBIRD_13_0b2_RELEASE REPO=releases/comm-beta PRODUCT=thunderbird APP_NAME=mail EMAIL=release@mozilla.com KEYDIR=d:/2011-keys RELCONFIG=fake STAGE_HOST=stage.mozilla.org STAGE_USERNAME=tbirdbld SSH_KEY=$HOME/.ssh/tbirdbld_dsa cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone hg pull && hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make tb-all PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \ KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \ APP_NAME=$APP_NAME TAG=$TAG \ STAGE_USERNAME=$STAGE_USERNAME \ STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY
- 1st attempt failed trying to checkout the signing tools from the mofo private CVS repo
- catlee suggested changing the key used for cvs in ~/.ssh/config. Switched it to the default (id_dsa) and it worked fine.
- checked output log
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \ KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} \ APP_NAME=$APP_NAME TAG=$TAG \ STAGE_USERNAME=$STAGE_USERNAME \ STAGE_HOST=$STAGE_HOST SSH_KEY=$SSH_KEY
Updates
- patcher_build_tools failed
- filed bug 756666 - Thunderbird updates builder needs to build patcher tools from mozilla repo, not comm repo
- patched ReleaseUpdatesFactory locally to set mozRepoPath, and reconfiged
- forced builder again
- backupsnip failed with:
bash: /home/tbirdbld/bin/backupsnip: No such file or directory
- filed bug 756599 - thunderbird release 'updates' failed to run backupsnip
- To fix, did:
# ffxbld@aus3-staging chmod 755 /home/ffxbld
- Then as tbirdbld@aus3-staging
cd /opt/aus2/snippets/staging ~ffxbld/bin/backupsnip Thunderbird-13.0b2-build1-test ~ffxbld/bin/pushsnip Thunderbird-13.0b2-build1-test
- Then as cltbld@linux64-ix-slave03:
cd /builds/slave/tb-rel-comm-beta-updates/build/temp/thunderbird/13.0b1-13.0b2 rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' aus2/ tbirdbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Thunderbird-13.0b2-build1 bash /builds/slave/tb-rel-comm-beta-updates/tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest
- Then did the following to get the automation going again:
# cltbld@buildbot-master34 mysql -u buildbot2 -p -h buildbot-rw-vip.db.scl3.mozilla.com buildbot_schedulers update schedulers set state='{"remainingBuilders": [], "upstreamBuilders": ["release-comm-beta-updates"], "lastReset": 1336754341.071274, "lastCheck": 1336754341.071274}' where name='release-comm-beta-updates_done'; update schedulers set state='{"remainingBuilders": [], "upstreamBuilders": ["release-comm-beta-thunderbird_source", "release-comm-beta-linux_build", "release-comm-beta-linux_repack_complete", "release-comm-beta-linux64_build", "release-comm-beta-linux64_repack_complete", "release-comm-beta-win32_build", "release-comm-beta-win32_repack_complete", "release-comm-beta-macosx64_build", "release-comm-beta-macosx64_repack_complete", "release-comm-beta-updates"], "lastReset": 1336754341.0741279, "lastCheck": 1337318106}' where name='release-comm-beta-thunderbird_deliverables_ready'; update schedulers set state='{"remainingBuilders": ["release-comm-beta-push_to_mirrors"], "upstreamBuilders": ["release-comm-beta-push_to_mirrors", "release-comm-beta-updates"], "lastReset": 1336754341.080065, "lastCheck": 1336754341.080065}' where name='release-comm-beta-thunderbird_uptake_check';
Update verify
- bhearsum fired off the initial round of update_verify a little too soon (by his own admission), so some builders failed to find updates.
- 2nd round was fine
Reset reserved slaves
Check permissions / AV scan
- check permissions failed because it's looking for files owned by group 'firefox'
find /pub/mozilla.org/thunderbird/nightly/13.0b2-candidates/build1/ ! -group firefox ! -path \'*/contrib*\' | wc -l)
- ignored output for now
- filed bug 756661 - check_permissions fails because thunderbird release files are not owned by group firefox
Push to internal mirrors
- automation failed because it was trying to create release dir as ffxbld
ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org mkdir -p /pub/mozilla.org/thunderbird/releases/13.0b2/
- re-ran the step by hand on mv-moz2-linux-ix-slave09.build.mozilla.org with --ssh-user and --ssh-key set
[cltbld@mv-moz2-linux-ix-slave09 tb-rel-comm-beta-psh-mrrrs]$ scripts/scripts/release/push-to-mirrors.sh mozilla/production_config.py push '--extra-excludes=*.zip' '--extra-excludes=*.zip.asc' --ssh-user tbirdbld --ssh-key tbirdbld_dsa
- proper solution is to make sure we point to the proper config (i.e. probably not mozilla/production_config.py)
- filed bug 756685 - push_to_mirrors fails for thunderbird when using ffxbld keys
Final verification
Record receipt of "completed_final_verification" emails (1 per platform)
Going to Beta
Run backupsnip
NOTE: Remember to do this at least 2 hours ahead of the expected "go to beta" email. details
# tbirdbld@aus3-staging.mozilla.org $ cd /opt/aus2/snippets/staging/ $ time ~/bin/backupsnip Thunderbird-13.0b2-build1
Done in 8 minutes
Push to beta
- Wait for "go" from release managers
- Publish (and record run time here): details
# tbirdbld@aus3-staging.mozilla.org $ cd /opt/aus2/snippets/staging/ $ time ~/bin/pushsnip Thunderbird-13.0b2-build1
- on completion, email r-d that VERSION updates are live on the beta channel
pushsnip ran in 4m 14s; email sent to r-d
Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.