Releases/Firefox 3.6.17/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

armenzg, rail - bug 649516

Signed-off Revision(s)

Tags

Build # Tag Changeset
1 FIREFOX_3_6_17_BUILD1 dd0bc929f34c
2 FIREFOX_3_6_17_BUILD2 dd0bc929f34c
3 FIREFOX_3_6_17_BUILD3 fab6db5c327d

Notifications

  • We have to notify AV vendors once the signed bits are on candidates directory (for email address and template example, see bug 568884#c15 )
  • We have to give mirrors as much notice as possible before pushing to mirrors. This has been 24-72 hours, but has been up to a week in advance. (for email address and template example, see bug 633657 )
  • We have to notify infra-all@ before "go to build" and before we ship. The more advance notice the better.
  • We have to notify metrics once we release

Bugs hit

  • bug 650305 - I forgot to take L10n changes; a build#2 had to be generated
  • I had to wait for keymaster to be free as there were two releases being signed already
  • bug 651802 - create_buildN_snippets fails for canceled previous builds
  • AUS config bump didn't land before beta, so beta users weren't getting updated to 3.5.19
  • bug 653676 - Forgot to unthrottle the current version and throttle 4.0.1. 3.6.16 people are not getting 3.6.17 because it is still throttled. Anyone who does get 3.6.17 is getting prompted with a 4.0.1 major update offer because it is not throttled.

NOTE: DON'T FORGET TO UNTHROTTLE BETA CHANNEL ON BETA DAY AND REMOVE IT ON THE RELEASE DAY ADDING THROTTLING FOR THE LAST VERSION ON THE RELEASE CHANNEL TO PREVENT UNPROMPTED MAJOR UPDATES!

See also: Post Mortem Notes

Build data

Build # Type Build ID Build machine Time to build
1 Linux canceled for build#2
Mac canceled for build#2
Windows canceled for build#2
2 Linux 20110414082317 mv-moz2-linux-ix-slave04 40 mins, 7 secs
Mac 20110414082317 bm-xserve16 1 hrs, 38 mins, 48 secs
Windows 20110414082317 mw32-ix-slave11 1 hrs, 47 mins, 20 secs
3 Linux 20110420140830 mv-moz2-linux-ix-slave04 42 mins, 48 secs
Mac 20110420140830 bm-xserve16 1 hrs, 48 mins, 57 secs
Windows 20110420140830 mw32-ix-slave11 1 hrs, 48 mins, 13 secs

Build 1

  • Clobbered "Any master/1.9.2/Any builder"
  • NOTE: I forgot to take L10n changes so I had to do build#2
  • Pushed buildbot-configs 9fd17bd38f69 and merged to production branch
  • tagged using FIREFOX_3_6_17_BUILD1 and FIREFOX_3_6_17_RELEASE: buildbotcustom (tip of production-0.8), tools (tip of default), buildbot-configs (tip of production)
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • reserved 6 slaves:
echo 6 >reserved_slaves_pm03
  • kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
/builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
-u armenzg -V 3.6.17 --branch mozilla-1.9.2 --build-number 1 \
-c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

NOTE: I had to cancel and start a build2 because I forgot to take l10n-changesets changes.

Build 2

wget -O l10n-changesets_mozilla-1.9.2 https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx3.6.17
  • Pushed buildbot-configs changes for build2 - http://hg.mozilla.org/build/buildbot-configs/rev/364a37722eff
  • tagged using FIREFOX_3_6_17_BUILD2 and FIREFOX_3_6_17_RELEASE: buildbotcustom (tip of production-0.8), tools (tip of default), buildbot-configs (tip of production)
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • reserved 6 slaves:
echo 6 >reserved_slaves_pm03
  • kicked off automation with:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
/builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
-u armenzg -V 3.6.17 --branch mozilla-1.9.2 --build-number 2 \
-c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010

Tag

No problems.

Update Bouncer

N/A for build#2.

Source / XULRunner Source

No problems.

Build

No problems.

Start autosigning

NOTE: I had to wait for an RDP session to be available (3.5 and 4.0 went first).

Set up autosinging using instructions

L10n Repack

XULRunner Build

No problems.

Unittests

*** registerContentHandler(text/html,http://localhost:8888/%s,Foo handler)
INFO | automation.py | Application ran for: 0:24:52.766000
INFO | automation.py | Reading PID log: c:\docume~1\cltbld\locals~1\temp\tmpourtsspidlog
==> process 3780 launched child process 3352
INFO | automation.py | Checking for orphan process with PID: 3352
SUCCESS: The process with PID 3972 has been terminated.

command timed out: 1200 seconds without output
program finished with exit code 1
elapsedTime=2703.687000
  • release-mozilla-1.9.2-macosx_test mochitests:
41654 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_reactivate.html | Test timed out.
58023 ERROR TEST-UNEXPECTED-FAIL | /tests/layout/base/tests/test_bug450930.xhtml | scrolled-out invalidation should notify in subdoc

Signing

Followed CombinedSigning doc.

L10n Verify

Nothing relevant on the logs.

Updates

3.6* -> 3.6.17

Generation

No problems.

Update Verify
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.14/20110218124600/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.14/20110218124600/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.14/20110218124600/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.14/20110218124600/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/ml/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.13/20101203074432/Darwin_Universal-gcc3/te/betatest/update.xml?force=1

3.6.x -> 4.0 Major Updates

MU Generation

Hit 'force build' to trigger it.

  • The slave that took the job did not clobber - bug 650268
  • I forced it again but it would fail every time - bug 650291

I worked around bug 650291 by backing out the update verify commit to the tools repo
done by the first job and 'force build':
http://hg.mozilla.org/build/tools/rev/cd5728db1b57
backed out by:
http://hg.mozilla.org/build/tools/rev/015583a6b6fe

MU Verify
  • bug 650933 - Disable MU verification for as, ka & oc for Firefox 4
    • We can ignore these failures as those locales are not shipped on Firefox 4
    • TODO land CVS patches on bug 650933
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Linux_x86-gcc3/as/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Linux_x86-gcc3/ka/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Linux_x86-gcc3/oc/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Darwin_x86-gcc3-u-ppc-i386/as/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Darwin_x86-gcc3-u-ppc-i386/ka/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/Darwin_x86-gcc3-u-ppc-i386/oc/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/WINNT_x86-msvc/as/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/WINNT_x86-msvc/ka/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110414082317/WINNT_x86-msvc/oc/betatest/update.xml?force=1

Socorro product data updates

crash-stat admin: added Firefox 3.6.17 and Firefox 3.6.18pre.

Going to beta day!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip Firefox-3.6.17-build2-beta

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip Firefox-3.6.17-build2-beta

Build 3

  • Clobbered mozilla-1.9.2, Any builder
  • Manually clobbered rel-192-mu mv-moz2-linux-ix-slave14 (see bug 650268)
  • Pushed buildbot-configs 9f88fd2fd8db and merged to production branch
  • tagged using FIREFOX_3_6_17_BUILD3 and FIREFOX_3_6_17_RELEASE: buildbotcustom (tip of production-0.8), tools (tip of default), buildbot-configs (tip of production)
  • updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/configs, and /builds/buildbot/builder_master/tools to latest
  • Disabled update bump, update verify bump, MU update bump, MU update verify bump by applying the patch locally
  • reserved 6 slaves: 
echo 6 > reserved_slaves_pm03
  • Started automation:
PYTHONPATH=/builds/buildbot/builder_master:/builds/buildbot/builder_master/tools/lib/python python \
  /builds/buildbot/builder_master/tools/buildbot-helpers/release_sanity.py \
  -u rail -V 3.6.17 --branch mozilla-1.9.2 --build-number 3 \
  -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9010
  • Pushed a patch with manual bump of the update verify and MU update verify configs.

Tag

No problems.

Start autosigning

Set up autosinging using instructions

Update Bouncer

N/A for buildNumber > 1

Source / XULRunner Source

No problems.

Build

No problems.

L10n Repack

No problems.

XULRunner Build

No problems.

Unittests

TODO: review oranges

Signing

No problems.

L10n Verify

Nothing unusual in the logs.

Updates

3.6* -> 3.6.17

Generation
  • Disabled update bump, update verify bump, MU update bump, MU update verify bump by applying the patch locally
  • Pushed a patch with manual bump of the update verify and MU update verify configs.
  • create_buildN_snippets step turned red due to missing build1 binaries and 3.6.16 mac snippets for ml and te. Ignorable.
  • The builder turned red (false alarm!), see bug 651802 for the details.
  • Triggered update verify, permission and AV check steps manually.
  • Sent an email to the AV vendors list (it didn't triggered due to this failure)
Update Verify

mac and win32: green

Linux failed with the following: 

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6b3/20091115172547/Linux_x86-gcc3/gu-IN/betatest/update.xml?force=1

Checked manually, worked fine. Nothing to worry about.

3.6.x -> 4.0 Major Updates

MU Generation

No problems. Notice that MU update config bump and MU update verify config bump steps were disabled.

Accidentally pressed "Force build" (should have pressed MU update verify rebuilds) and stopped the build.

MU Verify
  • Stopped the first run due to wrong buildid in MU update verify configs. Instead of buildids of 3.6.17 build3 there were buildids of 3.6.17 build2. AUS2 returned buildN snippets for those builds, so 3.6.17 build3 was "upgraded" to 3.6.17 build3 instead of 4.0.1.
  • Fixed the configs here: http://hg.mozilla.org/build/tools/rev/1ebcc708d553
  • triggered builds. Mac failed. Clobbered manually, retriggered.
  • Set reserved slaves to 0
  • reverted the patch on the pm03 and reconfigured it.

Green!

Socorro product data updates

Done in build2

Going to beta day!

Do this in the morning before the rel-driver gives you the GO. 

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging/
~/bin/backupsnip Firefox-3.6.17-build3-beta

Push to Beta Channel

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/pushsnip Firefox-3.6.17-build3-beta
 real    22m0.445s
 user    0m7.136s
 sys     0m39.062s

Email mirrors about release day date

joduinn sent an email a day before for all versions.

Going live day!

Final checks before push & virus scan

Reran permission checks and antivirus builders with the properties set by the previous automatic run:

  • release_config: mozilla/release-firefox-mozilla-1.9.2.py
  • script_repo_revision: FIREFOX_3_6_17_RELEASE

Push to Mirrors

Forced push_to_mirrors builder with the following properties set:

  • release_config: mozilla/release-firefox-mozilla-1.9.2.py
  • script_repo_revision: FIREFOX_3_6_17_RELEASE

Push to community mirrors: 

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.16 with 3.6.17

Final Verification split by platform

Got some 404s:

  • pj-mirror01.mozilla.org, which looks not up to date
  • www.mirrorservice.org was syncing.

Push release channel snippets

#From a build slave you can reach aus2-staging
#cltbld@aus2-staging.mozilla.org
#This takes now a lot of time because of PHX datacenter
cd /opt/aus2/snippets/staging/
~/bin/backupsnip Firefox-3.6.17-build3

time ~/bin/pushsnip Firefox-3.6.17-build3
real    19m17.841s
user    0m6.217s
sys     0m33.795s

~/bin/backupsnip Firefox-3.6.17-build3-4.0.1-build1-MU
real    1m56.305s
user    0m0.367s
sys     0m1.770s

time ~/bin/pushsnip Firefox-3.6.17-build3-4.0.1-build1-MU
real    0m30.299s
user    0m0.182s
sys     0m1.049s

AUS Throttling

See bug 653676 for the details and patch needed to unthrottle 3.6.16 -> 3.6.17, and throttle 3.6.17 -> 4.0.1. This should be done after pushing release snippets.

Release

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-{,pre}releases.exclude
# exclude 3.6.16

Update symlinks 

#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.17 latest-3.6

Send the following mail to metrics@mozilla.com: 

Firefox,firefox,firefox,3.6.17,3.6
Firefox,firefox,firefox,3.6.18pre,3.6

(Next time the email should be sent at the beginning of build1)

XULRunner Signing

Signed with: 

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.2.17
BUILD=3
TAG=FIREFOX_3_6_17_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.17-candidates/build3/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.17/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added 

- xulrunner/releases/1.9.2.16

...to exclude the previous release.

Update XULRunner entries on wiki

Updated the links in these two wiki pages for the version change:

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.6.17/BuildNotes&oldid=311614"