Releases/Firefox 3.5.19/BuildNotes
Contents
- 1 Notes About Releasing
- 2 Build Engineers
- 3 Notifications
- 4 Bugs hit
- 5 Signed-off Revision(s)
- 6 L10N changesets
- 7 Tags
- 8 Build data
- 9 Notes
- 9.1 Build 1
- 9.1.1 Tag
- 9.1.2 Source
- 9.1.3 Build
- 9.1.4 L10n Repack
- 9.1.5 Sign
- 9.1.6 L10n Verify
- 9.1.7 Updates
- 9.1.8 Update Verify
- 9.1.9 Socorro product data updates
- 9.1.10 Update Bouncer
- 9.1.11 Major Update Generation
- 9.1.12 Unreserve slaves
- 9.1.13 Publish updates to beta channel
- 9.1.14 Add beta to AUS throttling exceptions
- 9.2 Build 2
- 9.2.1 Tag
- 9.2.2 Source
- 9.2.3 Build
- 9.2.4 L10n Repack
- 9.2.5 Sign
- 9.2.6 L10n Verify
- 9.2.7 Updates
- 9.2.8 Major Update Generation
- 9.2.9 Unreserve slaves
- 9.2.10 Going to beta day!
- 9.2.11 Email mirrors about release day date
- 9.2.12 Final checks before push
- 9.2.13 Virus check
- 9.2.14 Pre-push Checks
- 9.2.15 Push to mirrors
- 9.2.16 Final verify
- 9.2.17 AUS Throttling
- 9.2.18 Push updates to release channel
- 9.2.19 Release
- 9.2.20 XULRunner
- 9.2.21 XULRunner Source & Build
- 9.1 Build 1
Notes About Releasing
Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed
Build Engineers
coop - bug 649514
Notifications
- Notify AV vendors ASAP
- Notify metrics once we release
Bugs hit
- starting autosign failed the first time because RELCONFIG was not defined
- https://intranet.mozilla.org/Build:CombinedSigning#Define_Application_Variables covers this, but the build notes from the past few 3.5 releases don't include a step to set that var.
- Notified AV vendors manually due to issues described in https://bugzilla.mozilla.org/show_bug.cgi?id=568884#c25
- whimboo reported seeing a recurrence of bug 628829, but only on the test channels.
- Armen followed the example here to fix snippets for some locales for both 3.6.17 and 3.5.19.
- hg server went down: bug 651780. This caused the updates step for build 2 to fail.
- NB: AUS config bump didn't land before beta, so beta users weren't getting updated to 3.5.19
- bug 653676 - Forgot to unthrottle the current version and throttle 4.0.1. 3.5.18 people are not getting 3.5.19 because it is still throttled. Anyone who does get 3.5.19 is getting prompted with a 4.0.1 major update offer because it is not throttled.
- This happened for 3.6.17 too. Sections for updating AUS have been EXPLICITLY ADDED to both sets of build notes.
- due to copy-n-paste from build1 notes -> build2 notes, XR builds for build2 weren't actually signed before being rsync-ed for release. Had to be signed later and then rsync-ed again: bug 654103
- avoid this by not copying build notes between builds
Signed-off Revision(s)
Build 1: 1c37949defbe
L10N changesets
Tags
Build # | Tag | Changeset |
1 | FIREFOX_3_5_19_BUILD1 GECKO19119_2011041408_RELBRANCH | 5341366b7692 |
2 | FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD2 GECKO19119_2011041408_RELBRANCH | 6a8fcd90b366 |
Build data
Build # | Type | Build ID | Build machine | Time to build |
1 | Linux | 20110414083440 | linux-ix-slave14 | 32 mins, 35 secs |
Mac | 20110414083440 | bm-xserve15 | 1 hrs, 56 mins, 3 secs | |
Windows | 20110414083440 | mw32-ix-slave02 | 1 hrs, 19 mins, 22 secs | |
2 | Linux | 20110420144310 | mv-moz2-linux-ix-slave02 | 30 mins, 5 secs |
Mac | 20110420144310 | bm-xserve11 | 1 hrs, 47 mins, 13 secs | |
Windows | 20110420144310 | mw32-ix-slave06 | 1 hrs, 20 mins, 26 secs |
Notes
Done on pm01
Build 1
- set release clobberer for mozilla-1.9.1, release-mozilla-1.9.1-post_signing, and release-mozilla-1.9.1-major_update on pm01
- set reserved slaves to 6 on pm01
- Landed updated automation config
- Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD1
- Updated buildbot-configs/buildbotcustom/tools on pm01
- Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
- After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py localhost:9010
Tag
No problems
Source
No problems
Build
- Linux: no problems
- Win32: no problems
- Mac: no problems
L10n Repack
- Linux: no problems
- Win32:
- repack builder 2/6 timed out when calling clobberer (1200s). Triggered a rebuild from the waterfall.
- Mac: no problems
Sign
Set up signing for 3.5.19. It was done concurrently with 4.0.1(and later, 3.6.17) by using two RDP sessions, one for each autosign instance.
Removed en-US.xpi from candidates directory.
PRODUCT=firefox VERSION=3.5.19 BUILD=1 TAG=FIREFOX_3_5_19_RELEASE REPO='releases/mozilla-1.9.1' EMAIL=release@mozilla.com KEYDIR=d:/2010-keys RELCONFIG=mozilla/release-firefox-mozilla-1.9.1.py cd ~/hg-tools hg pull && hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
Signature verification succeeded.
L10n Verify
All platforms failed 'l10n metadiff' with what looks to be search engine ordering changes e.g.
diff -r firefox-3.5.19-build1/diffs/linux-i686.af.diff firefox-3.5.18-build1/diffs/linux-i686.af.diff 12669c12669 < 1,49d0 --- > 1,47d0 12679,12680c12679,12680 < < browser.contentHandlers.types.0.title=Bloglines < < browser.contentHandlers.types.0.uri=http://www.bloglines.com/login?r=/sub/%s --- > < browser.contentHandlers.types.0.title=Google > < browser.contentHandlers.types.0.uri=http://fusion.google.com/add?feedurl=%s 12683,12684d12682 < < browser.contentHandlers.types.2.title=Google < < browser.contentHandlers.types.2.uri=http://fusion.google.com/add?feedurl=%s
Updates
No problems
Update Verify
whimboo reported seeing a recurrence of bug 628829, but only on the test channels.
Armen followed the example here to fix snippets for some locales for both 3.6.17 and 3.5.19.
Socorro product data updates
crash-stat admin: added Firefox 3.5.19 and Firefox 3.5.20pre .
Update Bouncer
Added by automation, no problems.
Major Update Generation
Needed to force a rebuild because the MU config was still pointing at 4.0rc2. Commented the following 2 lines in process/factory.py, reconfigured and rebuilt:
#self.bumpPatcherConfig() #self.bumpVerifyConfigs()
Local changes discarded, the master reconfigured after MU updates builder was triggered.
Major Update Verification
Failed for the following dropped locales: as, ka, mn, oc. Expected.
Unreserve slaves
Reset reserved slaves to 0
Publish updates to beta channel
cd /opt/aus2/snippets/staging/ time ~/bin/backupsnip Firefox-3.5.19-build1-beta real 56m38.988s user 0m7.943s sys 0m40.626s time ~/bin/pushsnip Firefox-3.5.19-build1-beta real 29m48.032s user 0m9.151s sys 0m48.627s
Add beta to AUS throttling exceptions
CVS diff. Done after pushing to beta channel.
Build 2
- set release clobberer for mozilla-1.9.1, release-mozilla-1.9.1-post_signing, and release-mozilla-1.9.1-major_update on pm01
- set reserved slaves to 6 on pm01
- Landed updated automation config
- Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_19_RELEASE FIREFOX_3_5_19_BUILD2
- Updated buildbot-configs/buildbotcustom/tools on pm01
- Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 2 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
- After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u coop -V 3.5.19 --branch mozilla-1.9.1 --build-number 2 -c release-firefox-mozilla-1.9.1.py localhost:9010
Tag
No problems
Source
No problems
Build
- Linux: no problems
- Win32: no problems
- Mac: no problems
L10n Repack
- Linux: no problems
- Win32:
- repack builder 2/6 timed out when calling clobberer (1200s). Triggered a rebuild from the waterfall.
- Mac: no problems
Sign
Set up signing for 3.5.19. It was done concurrently with 3.6.17 build3 by using two RDP sessions, one for each autosign instance.
PRODUCT=firefox VERSION=3.5.19 BUILD=2 TAG=FIREFOX_3_5_19_RELEASE REPO='releases/mozilla-1.9.1' EMAIL=release@mozilla.com KEYDIR=d:/2010-keys RELCONFIG=mozilla/release-firefox-mozilla-1.9.1.py cd ~/hg-tools hg pull && hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG} USE_NEW=1
Signature verification succeeded.
L10n Verify
Same "failures" as Build 1.
Updates
- 1st attempt: hg server went down, causing step to fail: bug 651780
- 2nd attempt: failed to find snippets for kn, ml, and te for 3.5.18: bug 629256, bug 651802
Update Verify
- Linux:
- transient snippet errors:
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.5/20090615155223/Linux_x86-gcc3/fi/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1 FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5b4/20090423192432/Linux_x86-gcc3/es-MX/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
- Mac:
- kn, ml, te:
Binary files source/Firefox.app/Contents/MacOS/plugins/DefaultPlugin.plugin/Contents/MacOS/DefaultPlugin and target/Firefox.app/Contents/MacOS/plugins/DefaultPlugin.plugin/Contents/MacOS/DefaultPlugin differ Binary files source/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/updater and target/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/updater differ FAIL: binary files found in diff FAIL: check_updates returned failure for Darwin_Universal-gcc3 downloads/Firefox 3.5.18.dmg vs. downloads/Firefox 3.5.19.dmg: 1
- transient snippet errors:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5b4/20090423191946/Darwin_Universal-gcc3/de/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
- Win:
- transient snippet errors:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.1b1/20081007144708/WINNT_x86-msvc/hu/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
Major Update Generation
No problems
Major Update Verification
- No snippets for as, ka, mn, oc on betatest
Unreserve slaves
Reset reserved slaves to 0
Going to beta day!
Do this in the morning before the rel-driver gives you the GO.
#From a build slave you can reach aus2-staging #cltbld@aus2-staging.mozilla.org cd /opt/aus2/snippets/staging/ time ~/bin/backupsnip Firefox-3.5.19-build2-beta
Push to Beta Channel
#From a build slave you can reach aus2-staging #cltbld@aus2-staging.mozilla.org #This takes now a lot of time because of PHX datacenter cd /opt/aus2/snippets/staging/ time ~/bin/pushsnip Firefox-3.5.19-build2-beta real 32m30.777s user 0m9.713s sys 0m49.973s
- abillings reported that users were not being updated on the beta channel: bug 652162
- AUS2 config changes needed to land earlier. Patch is here: https://bugzilla.mozilla.org/attachment.cgi?id=527860
Email mirrors about release day date
joduinn sent an email a day before for all versions.
Final checks before push
Virus check
re-built AV scan, no problems
Pre-push Checks
re-ran check permissions builder, no problems
Push to mirrors
Forced push_to_mirrors builder with the following properties set:
release_config: mozilla/release-firefox-mozilla-1.9.1.py script_repo_revision: FIREFOX_3_5_19_RELEASE
No problems.
Final verify
Was triggered automatically when there is enough uptake.
AUS Throttling
See bug 653676 for the details and patch needed to unthrottle 3.5.18 -> 3.5.19, and throttle 3.5.19 -> 4.0.1.
Push updates to release channel
#From a build slave you can reach aus2-staging #cltbld@aus2-staging.mozilla.org #This takes now a lot of time because of slow I/O in MPT cd /opt/aus2/snippets/staging/ ~/bin/backupsnip Firefox-3.5.19-build2
time ~/bin/pushsnip Firefox-3.5.19-build2 real 26m7.217s user 0m8.129s sys 0m42.729s
time ~/bin/backupsnip Firefox-3.5.19-build2-4.0.1-build1-MU real 1m41.181s user 0m0.266s sys 0m1.517s
time ~/bin/pushsnip Firefox-3.5.19-build2-4.0.1-build1-MU real 0m28.118s user 0m0.163s sys 0m0.979s
Release
# ffxbld@stage cd /home/ftp/pub/firefox/releases/ rm latest-3.5 && ln -s 3.5.19 latest-3.5
Sent the following to metrics@mozilla.com
Firefox,firefox,firefox,3.5.19,3.5 Firefox,firefox,firefox,3.5.20pre,3.5
FIREFOX DONE TO HERE
XULRunner
XULRunner Source & Build
- Source: no problems
- Linux: no problems
- Win32: no problems
- Mac: no problems
Signing
Signed with:
PRODUCT=xulrunner VERSION=1.9.1.19 BUILD=2 TAG=FIREFOX_3_5_19_RELEASE REPO="releases/mozilla-1.9.1" EMAIL=coop@mozilla.com KEYDIR=d:/2010-keys mkdir -p ~/signing-work/${PRODUCT}-${VERSION} cd ~/signing-work/${PRODUCT}-${VERSION} cd ~/hg-tools hg pull hg up -r ${TAG} cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ --exclude=*.txt --exclude=*-symbols.zip \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \ unsigned-build${BUILD} rsync -av --exclude=unsigned unsigned-build${BUILD}/ \ signed-build${BUILD}/ rsync -av unsigned-build${BUILD}/unsigned/ \ signed-build${BUILD}/ make checksum-files create-sigs stage verify-sigs \ PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \ REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rmdir signed-build${BUILD}/contrib{,-localized} rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ signed-build${BUILD}/ \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
Push XULRunner to mirrors
- xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \ /pub/mozilla.org/xulrunner/nightly/1.9.1.19-candidates/build2/ \ /pub/mozilla.org/xulrunner/releases/1.9.1.19/
Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added
- xulrunner/releases/1.9.1.18
...to exclude the previous release.
Update XULRunner wiki
Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!
XULRUNNER DONE TO HERE