Releases/Thunderbird 10.0.5esr/BuildNotes
From MozillaWiki
< Releases
Contents
- 1 Notes About Releasing
- 2 Bugs hit
- 3 Build Engineers
- 4 Signed-off Revision(s)
- 5 Build data
- 6 Notes
- 6.1 Build 1
- 6.2 Build 2
- 6.3 Build 3
- 6.3.1 Preparing to start Automation
- 6.3.2 E-mail Metrics
- 6.3.3 E-mail mirrors@mozilla.org
- 6.3.4 Edit rsync exclude files
- 6.3.5 Tag
- 6.3.6 Bouncer Submitter
- 6.3.7 Source
- 6.3.8 Build
- 6.3.9 Checksums
- 6.3.10 Updates
- 6.3.11 Reset reserved slaves
- 6.3.12 Check permissions / AV scan
- 6.3.13 Push files to internal mirrors
- 6.3.14 Final verification
- 6.4 Build 3 Spin 2
- 6.4.1 Preparing to start Automation
- 6.4.2 E-mail Metrics
- 6.4.3 E-mail mirrors@mozilla.org
- 6.4.4 Edit rsync exclude files
- 6.4.5 Tag
- 6.4.6 Bouncer Submitter
- 6.4.7 Source
- 6.4.8 Build
- 6.4.9 Checksums
- 6.4.10 Updates
- 6.4.11 Check permissions / AV scan
- 6.4.12 Push files to internal mirrors
- 6.4.13 Final verification
- 6.4.14 Backup snippets
- 6.4.15 Reset reserved slaves
- 6.4.16 Push to esr channel
- 6.4.17 Update symlinks
Notes About Releasing
Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed
Bugs hit
Enter any bugs pre-existing or newly discovered and filed during the release:
- bug 760353 - make source-package fails on comm-esr10 because MKDIR is undefined
- bug 760361 - l10n repacks failing in configure
-
bug 760385 - we clobber hg shares more frequently than necessarydupe of bug 748158 - bug 760389 - some builders not clobbered on build2
- bug 760390 - en-US builds not pulling right mozilla-esr10 revision
- bug 760397 - source builder not getting right mozilla-esr10 revision
- bug 760807 - Program Compatibility Assistant shown on Windows 7 when setting as default mail client (NSIS version)
- bug 761951 - not checking releasetest in final verification
- bug 763769 - release sanity complains about mozconfig differences
Build Engineers
nthomas - Tracking bug: bug 744599
Signed-off Revision(s)
L10N changesets: unchanged
Build1:
- comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/570c22432c13 on default
- mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
Build2:
- comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/48677c4f453a on GECKO1005_2012053119_RELBRANCH
- mozilla-esr10: http://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
Build 3:
- comm-esr10: https://hg.mozilla.org/releases/comm-esr10/rev/939269a86b88 on GECKO1005_2012053119_RELBRANCH
- mozilla-esr10: https://hg.mozilla.org/releases/mozilla-esr10/rev/5713c92407dd on GECKO1005_2012053118_RELBRANCH
Build data
Firefox
| Build # | Type | Build ID | Build machine | Time to build |
| 1 | Linux | 20120531193048 | mv-moz2-linux-ix-slave09 | 38 mins, 47 secs |
| Linux64 | bld-centos5-64-vmw-003 | 45 mins, 40 secs | ||
| Mac | moz2-darwin10-slave44 | 3 hrs, 32 mins, 31 secs | ||
| Windows | mw32-ix-slave26 | 1 hrs, 43 mins, 28 secs | ||
| 2 | Linux | 20120601013708 | mv-moz2-linux-ix-slave09 | 38 mins, 13 secs |
| 3 | Linux | 20120601041309 | mv-moz2-linux-ix-slave09 | 37 mins, 26 secs |
| Linux64 | bld-centos5-64-vmw-003 | 41 mins, 53 secs | ||
| Mac | moz2-darwin10-slave44 | 3 hrs, 32 mins, 5 secs | ||
| Windows | mw32-ix-slave02 | 1 hrs, 39 mins, 32 secs | ||
| 3 Spin 2 | Windows | 20120605154056 | mw32-ix-slave02 | 1 hrs, 39 mins, 18 secs |
Notes
Build 1
Preparing to start Automation
- first TB ESR release on Firefox infra so nothing to clobber
- Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:
[cltbld@buildbot-master34 master]$ pwd /builds/buildbot/build1/master [cltbld@buildbot-master43 master]$ cat reserved_slaves 6
- For build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)
# tbirdbld@stage cd /pub/mozilla.org/thunderbird/nightly/ version=10.0.5esr mkdir ../candidates/$version-candidates ln -s ../candidates/$version-candidates $version-candidates
- No l10n changes
- Double land automation configs
- Tag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD1}
- update and reconfigure the master
- dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
--branch comm-esr10 --build-number 1 \
--release-config release-thunderbird-comm-esr10.py --products thunderbird \
--dryrun -l localhost:9001
- start automation ← monitor progress on buildbot (e.g. comm-esr10)
- the same command as above but without the "--dryrun"
E-mail Metrics
Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.
Thunderbird,thunderbird,thunderbird,10.0.5,10.0 Thunderbird,thunderbird,thunderbird,10.0.5esr,10.0
E-mail mirrors@mozilla.org
N/A
Edit rsync exclude files
N/A
Tag
No problems
Bouncer Submitter
No problems
Source
Failed:
make source-package hg-bundle HG_BUNDLE_REVISION=7c765c397d25 in dir /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb (timeout 2700 secs) make[1]: Entering directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer' Packaging source tarball... p ../../mozilla/dist/source/ make[1]: p: Command not found make[1]: [source-package] Error 127 (ignored) (cd /builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10 && gtar -c --owner=0 --group=0 --numeric-owner --mode="go-w" --exclude=".hg*" --exclude="CVS" --exclude=".cvs*" --exclude=".mozconfig*" --exclude="*.pyc" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/Makefile" --exclude="/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/mozilla/dist" --exclude="objdir-tb" -f - ../comm-esr10) | bzip2 -vf > ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2 /bin/sh: ../../mozilla/dist/source/thunderbird-10.0.5esr.source.tar.bz2: No such file or directory gtar: Removing leading `../' from member names make[1]: *** [source-package] Error 1 make[1]: Leaving directory `/builds/slave/tb-rel-comm-esr10-thunderbird-source/comm-esr10/objdir-tb/mail/installer' make: *** [source-package] Error 2
bug 760353 filed.
Build
en-US
No problems.
Repacks
Failing in configure, bug 760361 filed on:
/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/configure Adding configure options from /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/.mozconfig: --with-l10n-base=/builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10 --enable-application=mail --enable-official-branding --enable-update-channel=esr --enable-update-packaging creating cache ./config.cache checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking build system type... x86_64-unknown-linux-gnu checking for gawk... gawk configure: error: Invalid value --with-l10n-base, /builds/slave/tb-rel-comm-esr10-lnx64-rpk-1/comm-esr10/obj-l10n/../../mozilla-esr10 doesn't exist
Standard8 ported some build changes over to comm-esr10, and we're respinning.
Build 2
Preparing to start Automation
- clobber Any master, release-comm-esr10 branch, any builder
- Double land config bump and l10n mozconfig fix.
- Retag buildbot-configs, buildbotcustom, & tools with build & release tags. (details)
- buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD2}
- buildbotcustom and tools, on build1 revision: hg tag THUNDERBIRD_10_0_5esr_BUILD2
- update and reconfigure the master
- dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
--branch comm-esr10 --build-number 2 \
--release-config release-thunderbird-comm-esr10.py --products thunderbird \
--dryrun -l localhost:9001
- start automation ← monitor progress on buildbot (e.g. comm-esr10)
- the same command as above but without the "--dryrun"
E-mail Metrics
N/A for build 2
E-mail mirrors@mozilla.org
N/A
Edit rsync exclude files
N/A
Tag
No problems.
Bouncer Submitter
N/A
Source
Failed to clobber, bug 760389. Manually clobbered and rebuilt, green.
Standard8 discovered it had the wrong mozilla-esr10 code - bug 760397.
Build
Also didn't clobber. Standard8 discovered that they have a gecko version of 10.0.6esrpre, so are getting source from mozilla-esr10's default branch - bug 760390
Another respin required.
Build 3
Preparing to start Automation
- clobber Any master, release-comm-esr10 branch, any builder
- do manual clobbers of source, *_build, *repack*
- Double land source builder fix and config bump
- Standard8 landed the mozconfig fix
- Retag buildbot-configs, buildbotcustom, & tools with build & release tags (details)
- buildbotcustom & buildbot-configs: hg tag -f THUNDERBIRD_10_0_5esr_{RELEASE,BUILD3}
- tools, on build2 revision: hg tag -r 82e252ae7d68 THUNDERBIRD_10_0_5esr_BUILD3
- update and reconfigure the master
- dry run automation start
cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas -V 10.0.5esr \
--branch comm-esr10 --build-number 3 \
--release-config release-thunderbird-comm-esr10.py --products thunderbird \
--dryrun -l localhost:9001
2012-06-01 03:51:57,136 : INFO : Branch comm-esr10 is enabled on master for thunderbird
2012-06-01 03:51:57,636 : INFO : Comparing thunderbird mozconfigs to nightly mozconfigs...
2012-06-01 03:51:58,037 : ERROR : found in mail/config/mozconfigs/macosx-universal/release but not in mail/config/mozconfigs/macosx-universal/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,037 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in macosx64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --with-macbundlename-prefix=Firefox', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --disable-install-strip']!
2012-06-01 03:51:58,038 : ERROR : found in mail/config/mozconfigs/macosx-universal/nightly but not in mail/config/mozconfigs/macosx-universal/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,469 : ERROR : found in mail/config/mozconfigs/win32/release but not in mail/config/mozconfigs/win32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,469 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']!
2012-06-01 03:51:58,470 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/release but not in mail/config/mozconfigs/linux64/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:58,754 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux64 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:58,754 : ERROR : found in mail/config/mozconfigs/linux64/nightly but not in mail/config/mozconfigs/linux64/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,058 : ERROR : found in mail/config/mozconfigs/linux32/release but not in mail/config/mozconfigs/linux32/nightly: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print --comm-rev=${COMM_REV} --mozilla-rev=${MOZILLA_REV}"
2012-06-01 03:51:59,059 : WARNING : mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print" not in linux ['ac_add_options --enable-update-channel=nightly', 'CXX=$REAL_CXX', 'CXX="ccache $REAL_CXX"', 'CC="ccache $REAL_CC"', 'mk_add_options PROFILE_GEN_SCRIPT=@TOPSRCDIR@/build/profile_pageloader.pl', 'ac_add_options --enable-codesighs', 'mk_add_options MOZ_MAKE_FLAGS="-j4"', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', 'ac_add_options --with-ccache=/usr/bin/ccache', 'export MOZILLA_OFFICIAL=1', 'export MOZ_TELEMETRY_REPORTING=1', "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py 10'"]!
2012-06-01 03:51:59,059 : ERROR : found in mail/config/mozconfigs/linux32/nightly but not in mail/config/mozconfigs/linux32/release: mk_add_options CLIENT_PY_ARGS="--hg-options='--verbose --time' --hgtool=../tools/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --skip-venkman --tinderbox-print"
2012-06-01 03:51:59,059 : ERROR : Error verifying mozconfigs
Which is fallout from the mozconfig change
- start automation wth -b to bypass error ← monitor progress on buildbot (e.g. comm-esr10)
- the same command as above but without the "--dryrun"
E-mail Metrics
N/A for build 3
E-mail mirrors@mozilla.org
N/A
Edit rsync exclude files
N/A
Tag
No problems
Bouncer Submitter
N/A
Source
No problems
Build
en-US
- linux32 ok
- linux64 and Mac failed in the compile step:
abort: unknown revision 'THUNDERBIRD_10_0_5esr_RELEASE'! Time: real 0.900 secs (user 0.467+0.000 sys 0.100+0.000) Traceback (most recent call last): File "/builds/slave/tb-rel-comm-esr10-osx64-bld/build/client.py", line 618, in <module> Executing command: ['python', '../tools/buildfarm/utils/hgtool.py', 'http://hg.mozilla.org/releases/mozilla-esr10/', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla'] Executing command: ['hg', 'update', '-r', 'THUNDERBIRD_10_0_5esr_RELEASE', '-R', '/builds/slave/tb-rel-comm-esr10-osx64-bld/build/mozilla', '--verbose', '--time']
which is related to bug 757798. Solution is to manually pull in /builds/hg-shared/releases/mozilla-esr10 on bld-centos5-64-vmw-003 and moz2-darwin10-slave44, then clobber the build dirs. Also do the pull on other slaves that have repo to avoid later (potential) bustage in l10n repacks). The rebuild linux64 and mac en-US (green)
- win32 failed on the same share issue; updated the share & manual clobber, then rebuild (green)
l10n repacks
No problems
Checksums
No problems
Updates
No problems
Update verify
No problems
Reset reserved slaves
No done, other releases using machines/master.
Check permissions / AV scan
No problems
Push files to internal mirrors
No problems
Final verification
No problems
bug 760807 will means we need to redo win32
Build 3 Spin 2
Preparing to start Automation
- rather than update the release config, move the tags etc etc etc when we are rebuilding windows only we do this file munging on ftp (hard linking):
# tbirdbld@stage
/pub/mozilla.org/thunderbird/candidates/10.0.5esr-candidates
cp -alv build3{,.pre-win32-rebuild}/
cd build3
rm -rf jsshell-win32.zip *SUMS* update/win32/ win32/ win32_info.txt
ie hard link as a quick backup, then remove win32 specific files and SUMS files that will need be regenerated
- manual clobber of tb-rel-comm-esr10* on mw32-ix-slave02 and 03
E-mail Metrics
N/A
E-mail mirrors@mozilla.org
N/A
Edit rsync exclude files
N/A
Tag
N/A
Bouncer Submitter
N/A
Source
N/A
Build
en-US
Used the rebuild button, no problems
l10n repacks
mw32-ix-slave10 didn't have makensis.exe renamed to makensisu-2.46.exe, so repack 6 was using 2.33 when configure ran. Job aborted, clobbered, nsis fixed, rebooted; sr removed from candidate dir; builder rebuilt.
All green after that.
Checksums
Rebuilt. Failed at the very end where it tried to recreate the contrib dirs which already existed. Harmless so ignored.
Updates
Backed out update verify configs, and rebuilt. No problems.
Update verify
No problems
Check permissions / AV scan
Both rebuilt. Permissions orange with:
WARN: target directory /pub/mozilla.org/thunderbird/releases/10.0.5esr/ exists
which is fine because that dir exists but is empty after emptying the earlier builds out.
A/V no problems.
Push files to internal mirrors
Failed because thunderbird/releases/10.0.5esr already existed. Removed this and rebuilt. Twice it turns out, but the first one succeeded ok.
Final verification
All green, but not actually checking the releasetest channel, so not verifying bouncer links OK - bug 761951. While writing the patch for that tested that win32 works OK on releasetest.
Backup snippets
time ~/bin/backupsnip Thunderbird-10.0.5esr-build3 real 0m37.623s
Reset reserved slaves
In this case it would be about evacuating the slaves from bm34, which is limited to T'bird releases again, but we'll need them for 14.0b1 anyway.
Push to esr channel
time ~/bin/pushsnip Thunderbird-10.0.5esr-build3
Update symlinks
#tbirdbld@stage cd /pub/mozilla.org/thunderbird/releases rm latest-10.0esr; ln -s 10.0.5esr latest-10.0esr
