Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

bug 760389 - Clobber doesn't work for some release builders
bug 762256 - buildbot-master34 has local changes
bug 762328 - Source step failed on Thunderbird 14.0b1
bug 762357 - moz2-darwin slaves still being used for mozilla-beta and comm-beta repacks
bug 762922 - improve signing client retry logic
bug 763027 - create_buildN_snippets step failed for all Thunderbird snippets for Darwin_x86_64-gcc3

Build Engineers

Build #1: kmoir
Build #2: coop
Tracking bug: bug 758758

Signed-off Revision(s)

Build 1:

comm-beta: http://hg.mozilla.org/releases/comm-beta/rev/709068332aaf 709068332aaf
mozilla-beta: http://hg.mozilla.org/releases/mozilla-beta/rev/e835d6524759 e835d6524759

Build 2:

caused by releng issues, code revs are the same

L10N changesets

Instructions on how to get them

Thunderbird: https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=tb14_beta_b1

Build data

Thunderbird

Build #	Type	Build ID	Build machine	Time to build
1	Linux		linux-ix-slave18	33 mins, 23 secs
	Linux64		linux64-ix-slave07	32 mins, 58 secs
	Mac		bld-lion-r5-067	1 hrs, 29 mins, 30 secs
	Windows		mw32-ix-slave02	1 hrs, 38 mins, 53 secs
2	Linux	20120607154537	linux-ix-slave18	34 mins, 23 secs
	Linux64		linux64-ix-slave06	33 mins, 48 secs
	Mac		bld-lion-r5-067	1 hrs, 16 mins, 51 secs
	Windows		mw32-ix-slave10	1 hrs, 34 mins, 26 secs

Notes

Build 1

Preparing to start Automation

detailed instructions

Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
- clobberer
Reserve slaves

[cltbld@buildbot-master34 master]$ echo "10" > /builds/buildbot/build1/master/reserved_slaves

For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# tbirdbld@stage
cd /pub/mozilla.org/thunderbird/nightly/
mkdir ../candidates/14.0b1-candidates
ln -s ../candidates/14.0b1-candidates 14.0b1-candidates

Update l10n changesets for desktop, mobile
- Instructions on how to get them
- dashboard
Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
- hg tag THUNDERBIRD_14_0b1_{RELEASE,BUILD2}
update and reconfigure the master

# this removed all local changes
cd /builds/buildbot/build1
make update && make checkconfig && python buildbot-wrangler.py reconfig `pwd`/master

start automation

cd /builds/buildbot/build1
source bin/activate
buildbot sendchange --username=kmoir --master localhost:9001 --branch=releases/comm-beta \
  --comments "THUNDERBIRD 14.0b1 BUILD 1" -p products:thunderbird \
  -p script_repo_revision:THUNDERBIRD_14_0b1_BUILD2 release_build

start automation ← monitor progress on buildbot (e.g. release-comm-beta)

Manual patch + reconfig

After several attempts at this (sans reconfig), we went with the

make update && make checkconfig && python buildbot-wrangler.py reconfig `pwd`/master

that inadvertently blew away local changes.

Aki logged into bm34, applied the .diff files

-rw-rw-r-- 1 cltbld cltbld 2291 May 12 20:29 buildbot-configs/disable-depnightly.diff
-rw-rw-r-- 1 cltbld cltbld  465 May 29 08:22 buildbot-configs/only-thunderbird-builds.diff
-rw-rw-r-- 1 cltbld cltbld 1941 Jun  5 09:09 buildbot-configs/tb13-final.diff
-rw-rw-r-- 1 cltbld cltbld  460 Jun  5 09:09 buildbotcustom/tb13-final.diff

in chronological order, via

cd buildbot-configs
patch -p1 < disable-depnightly.diff
# edited to deal with bitrot
patch -p1 < only-thunderbird-builds.diff
patch -p1 < tb13-final.diff # didn't re-apply the already-applied portion
hg diff > ../aki-configs.diff
cd ../buildbotcustom
patch -p1 < tb13-final.diff
cd ..
make checkconfig
make reconfig

Aki edited the /builds/buildbot/build1/Makefile to no longer |make update| by renaming the target to |update-will-wipe-local-changes|.

We then found an errant :1 at the end of the l10n-changesets (zh-TW), uncaught because we didn't run release-sanity (not in the previous beta's relnotes)
Removed, double-landed, re-tagged.
Aki tried to reconfig, then had to |make stop| and |make start|
Aki manually tagged/branched zh-TW

# cltbld@linux-ix-slave14
cd /builds/slave/rel-comm-beta-thunderbird-tag/zh-TW/
hg pull
hg out
hg update -C -r 74cf6875e7c5
hg branch
hg branch GECKO140_2012060613_RELBRANCH
hg branch
hg tag -r 74cf6875e7c5 -f THUNDERBIRD_14.0b1_BUILD1 THUNDERBIRD_14.0b1_RELEASE
hg out
hg push -e 'ssh -l tbirdbld -i ~cltbld/.ssh/tbirdbld_dsa' --new-branch ssh://hg.mozilla.org/releases/l10n/mozilla-beta/zh-TW

Handed off to nthomas to patch bm34, since there were a number of issues.
Reduced the local diff to:

[cltbld@buildbot-master34 build1]$ for r in buildbot-configs buildbotcustom tools/; do hg -R $r diff; echo; done
diff -r c3fba09cd5d5 mozilla/builder_master.cfg
--- a/mozilla/builder_master.cfg	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/builder_master.cfg	Wed Jun 06 15:03:37 2012 -0700
@@ -71,6 +71,10 @@
 import passwords
 reload(passwords)
 
+ACTIVE_BRANCHES = []
+ACTIVE_THUNDERBIRD_BRANCHES = []
+ACTIVE_PROJECTS = []
+
 for branch in ACTIVE_BRANCHES:
     branchObjects = generateBranchObjects(BRANCHES[branch], branch,
             getattr(passwords, 'secrets', None))
diff -r c3fba09cd5d5 mozilla/production_config.py
--- a/mozilla/production_config.py	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/production_config.py	Wed Jun 06 15:03:37 2012 -0700
@@ -5,7 +5,8 @@
                  ['linux-ix-slave%02i' % x for x in [1,2,6] + range(12,43)]
 LINUX64_VMS    = ['bld-centos5-64-vmw-%03i' % x for x in range(1, 7)]
 LINUX64_IXS    = ['linux64-ix-slave%02i' % x for x in range(3,22)]
-WIN32_IXS      = ['mw32-ix-slave%02i' % x for x in range(2,16) + [20, 26]]
+#WIN32_IXS      = ['mw32-ix-slave%02i' % x for x in range(2,16) + [20, 26]]
+WIN32_IXS      = ['mw32-ix-slave02', 'mw32-ix-slave05', 'mw32-ix-slave10']
 WIN64_IXS      = ['w64-ix-slave%02i' % x for x in range(6,22) + range(22,25) + range(64,85)]
 WIN64_METRO    = ['w64-ix-slave%02i' % x for x in [22,40,42]]
 MOCK_DL120G7   = ['bld-centos6-hp-%03d' % x for x in range(6,24)] # 5 staging, 17 prod, 17 try
diff -r c3fba09cd5d5 mozilla/release-thunderbird-comm-beta.py
--- a/mozilla/release-thunderbird-comm-beta.py	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/release-thunderbird-comm-beta.py	Wed Jun 06 15:03:37 2012 -0700
@@ -131,3 +131,5 @@
 # Misc configuration
 releaseConfig['enable_repo_setup'] = False
 releaseConfig['enableAutomaticPushToMirrors'] = True
+
+releaseConfig['skip_tag'] = 1

and reconfiged.

Cleaned up the slaves from the previous failed builds
cd /builds/slave
dirs=`ls -d tb-rel-comm-beta* | grep -v tag`; rm -rf $dirs
mxw32-ix-slave02
mxw32-ix-slave05
mxw32-ix-slave10
linux64-ix-slave07
linux-ix-slave14
linux-ix-slave16

sanity check the config

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
   --branch comm-beta --build-number 1 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   --dryrun -l localhost:9001

run the build
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
   --branch comm-beta --build-number 1 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   -b -l localhost:9001

June 7, 2012

Catlee fixed bug 762357 and I patched bm34 with this change. Then ran

cd /builds/buildbot/build1/master
  source ../bin/activate
  PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
    --branch comm-beta --build-number 2 \
    --release-config release-thunderbird-comm-beta.py --products thunderbird  \
    --dryrun -l localhost:9001

run the build
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
   --branch comm-beta --build-number 2 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   -m -l localhost:9001

Note that I also updated it to be build 2

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.

Thunderbird,thunderbird,thunderbird,14.0b1,14.0

Tag

See above for problems, ended up being run multiple times.

Bouncer Submitter

Source

Build

Thunderbird

The win32 build died with

Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-bld/build\\mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_14.0b1_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-bld/build\\mozilla', '--verbose', '--time']
abort: unknown revision 'THUNDERBIRD_14.0b1_RELEASE'!
Time: real 10.752 secs (user 7.500+0.000 sys 0.266+0.000)

which is bug 757798. Fix:

# mw32-ix-slave02
cd /e/builds/hg-shared/releases/mozilla-beta
hg pull

and used Windows Explorer to delete e:\builds\moz2_slave\tb-rel-comm-beta-w32-bld. Then used Rebuild button. Did the pull on mw32-ix-slave05 and 10 too to preempt problems with l10n repacks.

linux32 (linux-ix-slave18) and linux64 (linux64-ix-slave07) aslo failed out with the same revision issue. Similar fix

cd /builds/hg-shared/releases/mozilla-beta && hg pull
rm -rf /builds/slave/tb-rel-comm-beta-*bld

And preemptively do the same on linux-ix-slave{02,14,15,16} and linux64-ix-slave{15,17}.

Build 2

Preparing to start Automation

Due to local patches, I ran the following by hand:

# cltbld@buildbot-master34
cd /builds/buildbot/build1/buildbot-configs
hg pull && hg up -r production
cd /builds/buildbot/build1/buildbotcustom
hg pull && hg up -r production-0.8

Because tagging for build2 had already been done, added a "releaseConfig['skip_tag'] = 1" to release-thunderbird-comm-beta.py.

Run release_sanity.py:

# cltbld@buildbot-master34
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u coop -V 14.0b2 \
   --branch comm-beta --build-number 2 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   --dryrun -l localhost:9001

...and of course it's not clean, but part of that is the skip_tag local change:

2012-06-07 14:44:42,815 : CRITICAL : Tests Failed! Not running sendchange!
2012-06-07 14:44:42,815 : CRITICAL : Failed tests (run with -b to skip) :
2012-06-07 14:44:42,815 : CRITICAL : verify_mozconfig
2012-06-07 14:44:42,815 : CRITICAL : verify_configs
2012-06-07 14:44:42,815 : CRITICAL : verify_options

Force the sendchange with -b:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u coop -V 14.0b2 \
   --branch comm-beta --build-number 2 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird \
   -b -l localhost:9001 2>&1 | tee release_sanity.log

Tag

Skipped

Bouncer Submitter

Not run for build# > 1.

Source

first attempt: source and build factories died with "abort: unknown revision 'THUNDERBIRD_14.0b1_RELEASE'!" same as first pass of build1. Followed same platform-specific steps as per build1, which amount to running:

cd /builds/hg-shared/releases/mozilla-beta && hg pull
rm -rf /builds/slave/tb-rel-comm-beta-*bld

...on all build slaves, modulo path differences on Windows.

Re-ran release-_sanity.py with -b, but had to reboot the linux slaves before the reconfig would complete.

second attempt: no problems

Build

first attempt: failed, see Source above
second attempt: No problems.

l10n repacks

Linux & Windows: No problems.
Mac: only 1 lion slave was attached to the master, which would have been sufficient if we hadn't hit bug 762922 which caused signing to fail for some locales once the token expired due to broken retry logic.
- added 4 more lion slaves to the master temporarily.
- bhearsum removed mac-signing2 from the pool of signing machines.
- killed the in-progress l10n mac repack jobs (4) because they were likely to fail anyway
- restarted the master (reconfigs hang indefinitely for some reason)
- pressed 'rebuild' on the 4 repack chunk builders I had stopped before.
- used the standalone_repack builder to recreate the failed locales from the 2 chink builders that had run previously.
- pressed 'force build' on the macosx64 repacks_complete builder once all the repacks were completed successfully

Updates

bug 763027 - create_buildN_snippets step failed for all Thunderbird snippets for Darwin_x86_64-gcc3
- not critical, builder continues

Update verify

No problems.

Reset reserved slaves

[cltbld@buildbot-master34 ~]$ echo "0" > /builds/buildbot/build1/master/reserved_slaves

Check permissions / AV scan

AV: OK. Permission check: failed (see bugs hit list)

Push to internal mirrors

Final verification

Going to Beta

Run backupsnip

# tbirdbld@aus3-staging
~/bin/backupsnip Thunderbird-14.0b1-build2

Push to beta

# tbirdbld@aus3-staging
~/bin/pushsnip Thunderbird-14.0b1-build2

DONE TO HERE

Build #	Branch, Tags	Changeset
1	GECKO140_2012060613_RELBRANCH THUNDERBIRD_14_0b1_BUILD1 THUNDERBIRD_14_0b1_RELEASE	9e95fa2b5145
1	GECKO140_2012060511_RELBRANCH THUNDERBIRD_14_0b1_BUILD1 THUNDERBIRD_14_0b1_RELEASE	dc3eeed0cb1c
2	THUNDERBIRD_14_0b1_BUILD2	2dce0acae7e5
2	THUNDERBIRD_14_0b1_BUILD2	aa4f19336ab4

Releases/Thunderbird 14.0b1/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Thunderbird

Notes

Build 1

Preparing to start Automation

Manual patch + reconfig

E-mail Metrics

Tag

Bouncer Submitter

Source

Build

Thunderbird

Build 2

Preparing to start Automation

Tag

Bouncer Submitter

Source

Build

l10n repacks

Updates

Update verify

Reset reserved slaves

Check permissions / AV scan

Push to internal mirrors

Final verification

Going to Beta

Run backupsnip

Push to beta

Navigation menu

Search