Releases/Firefox 11.0b3/BuildNotes
From MozillaWiki
< Releases
Contents
- 1 Notes About Releasing
- 2 Bugs hit
- 3 Build Engineers
- 4 Signed-off Revision(s)
- 5 L10N changesets
- 6 Tags
- 7 Build data
- 8 Notes
- 8.1 Build 1
- 8.2 Build 2
- 8.2.1 Preparing to start Automation
- 8.2.2 Tag
- 8.2.3 Bouncer Submitter
- 8.2.4 Source
- 8.2.5 Build
- 8.2.6 Unittests / Talos
- 8.2.7 XULRunner
- 8.2.8 Android Signing
- 8.2.9 L10N verify
- 8.2.10 Checksums
- 8.2.11 Updates
- 8.2.12 Reset reserved slaves
- 8.2.13 Check permissions / AV scan
- 8.2.14 Going to Beta
- 8.2.15 Final verification
- 8.2.16 Release Day
Notes About Releasing
Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed
Bugs hit
Enter any bugs pre-existing or newly discovered and filed during the release:
- whitelist bugs:
- bug 687179 - tagging should use one tagging operation per tagged revision
- bug 607392 - split tagging into en-US and other (broke all l10n repacks)
- bug 728109 - zip files published into releases/11.0b3/win32/*/
- regression from bug 726508 - no asc files except at the top level
- bug 728763 - Missing Xulrunner 11.0b3 For Windows
Build Engineers
aki, nthomas - Tracking bug: bug 718900
Signed-off Revision(s)
Build 1: 805226ae6764
Build 2: f21c6aa0f8c2
L10N changesets
Instructions on how to get them
- Firefox: l10n-changesets
- Fennec: No signed off revisions
Tags
| Build # | Branch, Tags | Changeset |
| 1 | MOBILE110_2012021420_RELBRANCH, FENNEC_11_0b3_RELEASE, FENNEC_11_0b3_BUILD1 | aa23dd738055 |
| GECKO110_2012021420_RELBRANCH, FIREFOX_11_0b3_BUILD1 FIREFOX_11_0b3_RELEASE | 805226ae6764 | |
| 2 | MOBILE110_2012021522_RELBRANCH, FENNEC_11_0b3_RELEASE, FENNEC_11_0b3_BUILD2 | 2427757522ff |
| GECKO110_2012021522_RELBRANCH, FIREFOX_11_0b3_BUILD2 FIREFOX_11_0b3_RELEASE | bb27964243e7 |
Build data
Firefox
| Build # | Type | Build ID | Build machine | Time to build |
| 1 | Linux | 20120214205459 | mv-moz2-linux-ix-slave10 | 1 hrs, 3 mins, 28 secs |
| Linux64 | moz2-linux64-slave06 | 3 hrs, 57 mins, 35 secs | ||
| Mac | moz2-darwin10-slave41 | 3 hrs, 14 mins, 54 secs | ||
| Windows | mw32-ix-slave07 | 3 hrs, 16 mins, 13 secs | ||
| 2 | Linux | 20120215222917 | mv-moz2-linux-ix-slave10 | 1 hrs, 8 mins, 7 secs |
| Linux64 | moz2-linux64-slave06 | 4 hrs, 3 mins, 14 secs | ||
| Mac | moz2-darwin10-slave56 | 3 hrs, 9 mins, 55 secs | ||
| Windows | mw32-ix-slave07 | 3 hrs, 18 mins, 28 secs |
Fennec
| Build # | Type | Build ID | Build machine | Time to build |
| 1 | android-xul | 20120214203536 | mv-moz2-linux-ix-slave08 | 47 mins, 42 secs |
| 2 | android-xul | 20120215221354 | mv-moz2-linux-ix-slave08 | 50 mins, 58 secs |
Notes
Build 1
Preparing to start Automation
- Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
- Reserve slaves
- Update l10n changesets for desktop, mobile
- Landed configs, Rail's win32 l10n signing fix, mozharness android signing fix
- Tag buildbot-configs, buildbotcustom, & tools with build & release tags
hg tag -f FIREFOX_11_0b3_BUILD1 FIREFOX_11_0b3_RELEASE FENNEC_11_0b3_BUILD1 FENNEC_11_0b3_RELEASE
- update and reconfigure the master
- start automation
cd master . ../bin/activate PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u aki -pfirefox,fennec \ -V 11.0b3 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py \ --dryrun localhost:9001 2>&1 | tee aki.out
Verified the mozconfig diffs were in the above bugs; kicked off automation with -b:
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u aki -pfirefox,fennec \ -V 11.0b3 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py \ -b localhost:9001 2>&1 | tee aki.out
E-mail Metrics
"metrics-alerts < AT > mozilla < PERIOD > org"
Firefox,firefox,firefox,11.0b3,11.0 Fennec,firefox-mobile,fennec,11.0b3,11.0
Tag
Hit bug 687179; Rail landed a fix and tagged.
- Re-set clobber
- Re-ran release sanity
Bouncer Submitter
No problems.
Source
No problems.
- Firefox: mv-moz2-linux-ix-slave05
- Fennec: mv-moz2-linux-ix-slave18
Build
Firefox
No problems.
Firefox repacks
- All repacks failed on script_repo_revision not being set; I currently suspect bug 607392.
- Forcing a repack with script_repo_revision set to FIREFOX_11_0b3_RELEASE failed later.
- Trying again with script_repo_revision set to FIREFOX_11_0b3_RELEASE and branch set to releases/mozilla-beta.
- That worked for some linux repacks, but failed on windows and mac. Nick pointed out that clobberer seems to be broken for repacks since the branch is being set to None.
- Rail backed out the offending patch and I reconfiged.
- Rerunning the remaining repacks seemed to work at this point, though the branch is still set to None.
- Win32 didn't send out an 'all win32 builds available' email so I forced that builder too.
Fennec
Android-xul build failed cloning mozharness but retried automatically:
retry: Calling <function run_with_timeout at 0xb7cc7fb4> with args: (['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'], 1260, None, None, False, True), kwargs: {}, attempt #1
Executing: ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness']
requesting all changes
adding changesets
adding manifests
adding file changes
added 680 changesets with 1291 changes to 173 files (+1 heads)
abort: data/.hgtags.i@f2b4bbef73df: no match found!
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0xb7cc7fb4> with args: (['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'], 1260, None, None, False, True), kwargs: {}, attempt #2
Executing: ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness']
abort: destination 'mozharness' is not empty
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0xb7cc7fb4> with args: (['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'], 1260, None, None, False, True), kwargs: {}, attempt #3
Executing: ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness']
abort: destination 'mozharness' is not empty
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0xb7cc7fb4> with args: (['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'], 1260, None, None, False, True), kwargs: {}, attempt #4
Executing: ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness']
abort: destination 'mozharness' is not empty
retry: Failed, sleeping 1 seconds before retrying
retry: Calling <function run_with_timeout at 0xb7cc7fb4> with args: (['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'], 1260, None, None, False, True), kwargs: {}, attempt #5
Executing: ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness']
abort: destination 'mozharness' is not empty
retry: Giving up on <function run_with_timeout at 0xb7cc7fb4>
Unable to successfully run ['hg', 'clone', 'http://hg.mozilla.org/build/mozharness', 'mozharness'] after 5 attempts
Retry worked.
Fennec repacks
None.
Unittests / Talos
XULRunner
Source
No problems.
Build
No problems.
Signing
TODO
Follow instructions in XULRunner Signing (aut required).
Android Signing
Instructions in MobileSigning
No problems.
Android Signature Verification
No problems.
L10N verify
Kicked off by hand.
Checksums
Kicked off by hand.
No problems.
Updates
Kicked off by hand.
RESPIN ANNOUNCED.
Build 2
Preparing to start Automation
- Set clobbers for the appropriate masters.
- Reserve slaves - left it from build 1.
- No l10n changesets for chemspill.
- Landed configs,
- Tag buildbot-configs, buildbotcustom, & tools with build & release tags
hg tag -f FIREFOX_11_0b3_BUILD2 FIREFOX_11_0b3_RELEASE FENNEC_11_0b3_BUILD2 FENNEC_11_0b3_RELEASE
- update and reconfigure the master
- start automation
cd master . ../bin/activate PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u aki -pfirefox,fennec \ -V 11.0b3 --branch mozilla-beta --build-number 2 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py \ --dryrun localhost:9001 2>&1 | tee aki.out
Verified the mozconfig diffs were in the above bugs; kicked off automation with -b:
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u aki -pfirefox,fennec \ -V 11.0b3 --branch mozilla-beta --build-number 2 -c release-firefox-mozilla-beta.py -c release-fennec-mozilla-beta.py \ -b localhost:9001 2>&1 | tee aki.out
Tag
No problems.
- Fennec: mv-moz2-linux-ix-slave18
- Firefox: mv-moz2-linux-ix-slave10
Bouncer Submitter
N/A
Source
No problems
Build
Firefox
No problems
Firefox repacks
No problems
Fennec
No problems with android-xul
Fennec repacks
N/A
Unittests / Talos
XULRunner
Source
No problems
Build
No problems
Signing
Updated sign_xulrunner.sh and ran it (XULRunner instructions).
Android Signing
Instructions in MobileSigning
# cltsign@signing1 cd signing-work/mozharness hg pull -u cd .. python2.6 mozharness/scripts/sign_android.py --config-file signing/android_mozilla-beta.py
Android Signature Verification
No problems
L10N verify
No problems.
Checksums
No problems.
Updates
Update didn't trigger automatically. Scheduler database was in a weird state:
| 3746 | release-mozilla-beta-firefox_deliverables_ready | buildbotcustom.scheduler.AggregatingScheduler | {"remainingBuilders": ["release-mozilla-beta-firefox_source", "release-mozilla-beta-linux_build", "release-mozilla-beta-linux_repack_complete"], "lastReset": 1329372616.24244, "lastCheck": 1329404507.84547} |
| 3752 | release-mozilla-beta-firefox_signing_done | buildbotcustom.scheduler.AggregatingScheduler | {"remainingBuilders": ["release-mozilla-beta-linux_build", "release-mozilla-beta-linux_repack_complete"], "lastReset": 1329372616.224206, "lastCheck": 1329404507.889406}
Worked around:
sql> UPDATE schedulers SET state='{"remainingBuilders": ["release-mozilla-beta-linux_repack_complete"], "lastReset": 1329372616.224206, "lastCheck": 1329405704.6713779}' WHERE name='release-mozilla-beta-firefox_signing_done';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
sql> UPDATE schedulers SET state='{"remainingBuilders": ["release-mozilla-beta-linux_repack_complete"], "lastReset": 1329372616.24244, "lastCheck": 1329406064.67486}' WHERE name='release-mozilla-beta-firefox_deliverables_ready';
Then retriggered release-mozilla-beta-linux_repack_complete (the only remaining builder after sql update)
Update verify
A number of these failed, checking 11.0b2 instead of 11.0b3.
Rail landed a fix and we re-kicked these.
Killed a number to speed up 10.0.2.
Re-kicked after 10.0.2 pushed to mirrors; these passed.
Reset reserved slaves
Set to 0.
Check permissions / AV scan
Triggered. Check permissions passed. Killed AV after it passed Windows and Mac.
Going to Beta
Run backupsnip
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip Firefox-11.0b3-build1
Push to beta
Push files to internal mirrors
Started but killed for 10.0.2.
Restarted after 10.0.2 pushed to mirrors. I had to remove the existing releases/11.0b3 directory and restart.
Final verification
Failed for win32; rekicked after mirrors absorbed enough to work.
That worked.
Release Day
Publish Fennec to the Android Market
Push the files
- edited and used this script.
Android Market
Pushed.
Push XULRunner to Mirrors
# xrbld@stage rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \ --exclude=*.checksums --exclude=jsshell* \ /pub/mozilla.org/xulrunner/nightly/11.0b3-candidates/build2/ \ /pub/mozilla.org/xulrunner/releases/11.0b3/
