Revision as of 02:16, 7 June 2012

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

bug 760389 TB ESR source & en-US builders failed to clobber on build2
bug 762256 Bug 762256 - buildbot-master34 has local changes
bug 762328 - Source step failed on Thunderbird 14.0b1

Build Engineers

kmoir - Tracking bug: bug 758758

Signed-off Revision(s)

Build 1:

comm-beta: http://hg.mozilla.org/releases/comm-beta/rev/709068332aaf 709068332aaf
mozilla-beta: http://hg.mozilla.org/releases/mozilla-beta/rev/e835d6524759 e835d6524759

L10N changesets

Instructions on how to get them

Thunderbird: https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=tb14_beta_b1

Build data

Thunderbird

Build #	Type	Build ID	Build machine	Time to build
1	Linux		linux-ix-slave18	33 mins, 23 secs
	Linux64		linux64-ix-slave07	32 mins, 58 secs
	Mac		bld-lion-r5-067	1 hrs, 29 mins, 30 secs
	Windows		mw32-ix-slave02	1 hrs, 38 mins, 53 secs

Notes

Build 1

Preparing to start Automation

detailed instructions

Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
- clobberer
Reserve slaves

[cltbld@buildbot-master34 master]$ echo "10" > /builds/buildbot/build1/master/reserved_slaves

For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# tbirdbld@stage
cd /pub/mozilla.org/thunderbird/nightly/
mkdir ../candidates/14.0b1-candidates
ln -s ../candidates/14.0b1-candidates 14.0b1-candidates

Update l10n changesets for desktop, mobile
- Instructions on how to get them
- dashboard
Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)
Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
- hg tag THUNDERBIRD_14_0b1_{RELEASE,BUILD1}
update and reconfigure the master

# this removed all local changes
cd /builds/buildbot/build1
make update && make checkconfig && python buildbot-wrangler.py reconfig `pwd`/master

start automation

cd /builds/buildbot/build1
source bin/activate
buildbot sendchange --username=kmoir --master localhost:9001 --branch=releases/comm-beta \
  --comments "THUNDERBIRD 14.0b1 BUILD 1" -p products:thunderbird \
  -p script_repo_revision:THUNDERBIRD_14_0b1_BUILD1 release_build

start automation ← monitor progress on buildbot (e.g. release-comm-beta)

Manual patch + reconfig

After several attempts at this (sans reconfig), we went with the

make update && make checkconfig && python buildbot-wrangler.py reconfig `pwd`/master

that inadvertently blew away local changes.

Aki logged into bm34, applied the .diff files

-rw-rw-r-- 1 cltbld cltbld 2291 May 12 20:29 buildbot-configs/disable-depnightly.diff
-rw-rw-r-- 1 cltbld cltbld  465 May 29 08:22 buildbot-configs/only-thunderbird-builds.diff
-rw-rw-r-- 1 cltbld cltbld 1941 Jun  5 09:09 buildbot-configs/tb13-final.diff
-rw-rw-r-- 1 cltbld cltbld  460 Jun  5 09:09 buildbotcustom/tb13-final.diff

in chronological order, via

cd buildbot-configs
patch -p1 < disable-depnightly.diff
# edited to deal with bitrot
patch -p1 < only-thunderbird-builds.diff
patch -p1 < tb13-final.diff # didn't re-apply the already-applied portion
hg diff > ../aki-configs.diff
cd ../buildbotcustom
patch -p1 < tb13-final.diff
cd ..
make checkconfig
make reconfig

Aki edited the /builds/buildbot/build1/Makefile to no longer |make update| by renaming the target to |update-will-wipe-local-changes|.

We then found an errant :1 at the end of the l10n-changesets (zh-TW), uncaught because we didn't run release-sanity (not in the previous beta's relnotes)
Removed, double-landed, re-tagged.
Aki tried to reconfig, then had to |make stop| and |make start|
Aki manually tagged/branched zh-TW

# cltbld@linux-ix-slave14
cd /builds/slave/rel-comm-beta-thunderbird-tag/zh-TW/
hg pull
hg out
hg update -C -r 74cf6875e7c5
hg branch
hg branch GECKO140_2012060613_RELBRANCH
hg branch
hg tag -r 74cf6875e7c5 -f THUNDERBIRD_14.0b1_BUILD1 THUNDERBIRD_14.0b1_RELEASE
hg out
hg push -e 'ssh -l tbirdbld -i ~cltbld/.ssh/tbirdbld_dsa' --new-branch ssh://hg.mozilla.org/releases/l10n/mozilla-beta/zh-TW

Handed off to nthomas to patch bm34, since there were a number of issues.
Reduced the local diff to:

[cltbld@buildbot-master34 build1]$ for r in buildbot-configs buildbotcustom tools/; do hg -R $r diff; echo; done
diff -r c3fba09cd5d5 mozilla/builder_master.cfg
--- a/mozilla/builder_master.cfg	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/builder_master.cfg	Wed Jun 06 15:03:37 2012 -0700
@@ -71,6 +71,10 @@
 import passwords
 reload(passwords)
 
+ACTIVE_BRANCHES = []
+ACTIVE_THUNDERBIRD_BRANCHES = []
+ACTIVE_PROJECTS = []
+
 for branch in ACTIVE_BRANCHES:
     branchObjects = generateBranchObjects(BRANCHES[branch], branch,
             getattr(passwords, 'secrets', None))
diff -r c3fba09cd5d5 mozilla/production_config.py
--- a/mozilla/production_config.py	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/production_config.py	Wed Jun 06 15:03:37 2012 -0700
@@ -5,7 +5,8 @@
                  ['linux-ix-slave%02i' % x for x in [1,2,6] + range(12,43)]
 LINUX64_VMS    = ['bld-centos5-64-vmw-%03i' % x for x in range(1, 7)]
 LINUX64_IXS    = ['linux64-ix-slave%02i' % x for x in range(3,22)]
-WIN32_IXS      = ['mw32-ix-slave%02i' % x for x in range(2,16) + [20, 26]]
+#WIN32_IXS      = ['mw32-ix-slave%02i' % x for x in range(2,16) + [20, 26]]
+WIN32_IXS      = ['mw32-ix-slave02', 'mw32-ix-slave05', 'mw32-ix-slave10']
 WIN64_IXS      = ['w64-ix-slave%02i' % x for x in range(6,22) + range(22,25) + range(64,85)]
 WIN64_METRO    = ['w64-ix-slave%02i' % x for x in [22,40,42]]
 MOCK_DL120G7   = ['bld-centos6-hp-%03d' % x for x in range(6,24)] # 5 staging, 17 prod, 17 try
diff -r c3fba09cd5d5 mozilla/release-thunderbird-comm-beta.py
--- a/mozilla/release-thunderbird-comm-beta.py	Wed Jun 06 17:01:39 2012 -0400
+++ b/mozilla/release-thunderbird-comm-beta.py	Wed Jun 06 15:03:37 2012 -0700
@@ -131,3 +131,5 @@
 # Misc configuration
 releaseConfig['enable_repo_setup'] = False
 releaseConfig['enableAutomaticPushToMirrors'] = True
+
+releaseConfig['skip_tag'] = 1

and reconfiged.

Cleaned up the slaves from the previous failed builds
cd /builds/slave
dirs=`ls -d tb-rel-comm-beta* | grep -v tag`; rm -rf $dirs
mxw32-ix-slave02
mxw32-ix-slave05
mxw32-ix-slave10
linux64-ix-slave07
linux-ix-slave14
linux-ix-slave16

sanity check the config

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
   --branch comm-beta --build-number 1 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   --dryrun -l localhost:9001

run the build
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u kmoir -V 14.0b1 \
   --branch comm-beta --build-number 1 \
   --release-config release-thunderbird-comm-beta.py --products thunderbird  \
   -b -l localhost:9001

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.

Thunderbird,thunderbird,thunderbird,14.0b1,14.0

Tag

See above for problems, ended up being run multiple times.

Bouncer Submitter

Source

Build

Thunderbird

The win32 build died with

Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-beta', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-bld/build\\mozilla']
Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_14.0b1_RELEASE', '-R', 'e:/builds/moz2_slave/tb-rel-comm-beta-w32-bld/build\\mozilla', '--verbose', '--time']
abort: unknown revision 'THUNDERBIRD_14.0b1_RELEASE'!
Time: real 10.752 secs (user 7.500+0.000 sys 0.266+0.000)

which is bug 757798. Fix:

# mw32-ix-slave02
cd /e/builds/hg-shared/releases/mozilla-beta
hg pull

and used Windows Explorer to delete e:\builds\moz2_slave\tb-rel-comm-beta-w32-bld. Then used Rebuild button. Did the pull on mw32-ix-slave05 and 10 too to preempt problems with l10n repacks.

linux32 (linux-ix-slave18) and linux64 (linux64-ix-slave07) aslo failed out with the same revision issue. Similar fix

cd /builds/hg-shared/releases/mozilla-beta && hg pull
rm -rf /builds/slave/tb-rel-comm-beta-*bld

And preemptively do the same on linux-ix-slave{02,14,15,16} and linux64-ix-slave{15,17}.

@@ Line 56: / Line 56: @@
 |-
 | Mac
-|
+| bld-lion-r5-067
-|
+| 1 hrs, 29 mins, 30 secs
 |-
 | Windows
-|
+| mw32-ix-slave02
-|
+| 1 hrs, 38 mins, 53 secs
 |}

Build #	Branch, Tags	Changeset
1	GECKO130_2012XXXXXX_RELBRANCH THUNDERBIRD_14_0b1_BUILD1 THUNDERBIRD_14_0b1_RELEASE	X
1	GECKO130_2012XXXXXX_RELBRANCH THUNDERBIRD_14_0b1_BUILD1 THUNDERBIRD_14_0b1_RELEASE	X

Releases/Thunderbird 14.0b1/BuildNotes: Difference between revisions

Revision as of 02:16, 7 June 2012

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Thunderbird

Notes

Build 1

Preparing to start Automation

Manual patch + reconfig

E-mail Metrics

Tag

Bouncer Submitter

Source

Build

Thunderbird

Thunderbird repacks

Updates

Update verify

Reset reserved slaves

Check permissions / AV scan

Push to internal mirrors

Final verification

Going to Beta

Run backupsnip

Push to beta

Navigation menu

Releases/Thunderbird 14.0b1/BuildNotes: Difference between revisions

Revision as of 02:16, 7 June 2012

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Thunderbird

Notes

Build 1

Preparing to start Automation

Manual patch + reconfig

E-mail Metrics

Tag

Bouncer Submitter

Source

Build

Thunderbird

Thunderbird repacks

Updates

Update verify

Reset reserved slaves

Check permissions / AV scan

Push to internal mirrors

Final verification

Going to Beta

Run backupsnip

Push to beta

Navigation menu

Search