Releases/Firefox 3.6.26/BuildNotes

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• I landed my changes on default and forgot to merge to production before tagging the repos
  • release_sanity saved the day and I fixed it before starting anything
• bug 720743 - Linux repack 6/6 timed out
  • I hit "rebuild" and documented it
• bug 720761 - The bump of the version on default for a release triggers dependent builds
  • This does not affect the release but could suck several builders before the release jobs (build + tests happen on build machines for 3.6)
• Should we have the mirrors email lower in the build notes? I'm a week ahead from the release day but the email template says "hey, we're shipping in 24hours".
  • I guess just saying "24 hours" or "on this day" could make it unambiguous
  • could we automate this as well?
• bug 719436 update verify required some code back out - nthomas caught it
• I had to clean up the XulRunner signing documentation (see the changes)

Build Engineers

armenzg - Tracking bug: bug 719214

Signed-off Revision(s)

Build 1: 173bc943fe0d

L10N changesets

Instructions on how to get them

• Firefox: [1]
• Fennec: N/A

Tags

Build #	Branch, Tags	Changeset
	GECKO19226_2012012406_RELBRANCH FIREFOX_3_6_26_BUILD1 FIREFOX_3_6_26_RELEASE	982f9c134751

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120124074252	mv-moz2-linux-ix-slave04	40 mins, 5 secs
	Mac		bm-xserve11	2 hrs, 6 mins, 54 secs
	Windows		mw32-ix-slave03	1 hrs, 44 mins, 34 secs

Notes

Build 1

Preparing to start Automation

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
  • "any master", "1.9.2", "any builder"
• Reserve slaves

echo 10 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

• Update l10n changesets for desktop, mobile
• Land automation configs
• Tag buildbot-configs, buildbotcustom, & tools with build & release tags
  • FIREFOX_3_6_26_{RELEASE,BUILD1}
• update and reconfigure the master
  • cd /builds/buildbot/build1
  • make update && make reconfigure
• start automation

 cd /builds/buildbot/build1/master
 source ../bin/activate
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001

The real thing:

cd /builds/buildbot/build1/master
 source ../bin/activate
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

• Sent to "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,3.6.26,3.6
Firefox,firefox,firefox,3.6.27pre,3.6

E-mail mirrors@mozilla.org

XXX: TODO They want to know approximately what time push to external mirrors will occur. Ideally this is a 24hr notice to them, with the estimated time.

Subject: Firefox {VERSION} coming to mirrors tomorrow
Body:
We're planning on pushing our Firefox {VERSION} release to mirrors sometime over 
the next 24 hours in time.
# If this is a chemspill, please mention that in the email as it will mean pinging 
# for CDN to be enabled when the push to internal mirrors happens

If you believe these releases or these dates will cause any problems,
please notify release at mozilla.com. If you have any problems with mirror
status/weights/etc, please notify mirror-submissions at mozilla.org.

Tag

No problems.

Bouncer Submitter

No problems.

Source

No problems.

Start autosign

NOTE: use new signcode keys from d:/2011-keys

Instructions are in CombinedSigning

Build

Firefox

No problems.

Firefox repacks

• Hit bug 720743 where one of the Linux repacks timed out when cloning mozilla-1.9.2
  • I hit "rebuild" and carried on

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

From release-mozilla-1.9.2-linux-opt-unittest-xpcshell-build9.txt.gz:

TEST-UNEXPECTED-FAIL | /builds/slave/test/build/xpcshell/tests/xpcom/unit/test_nsIProcess.js | test failed (with xpcshell return code: 0), see following log:

• A second run passed
• This is an intermittent orange (see bug 493778)

From release-mozilla-1.9.2-win32-opt-unittest-mochitest-other-build10.txt.gz:

TEST-UNEXPECTED-FAIL | chrome://mochikit/content/browser/browser/components/places/tests/perf/browser_ui_history_sidebar.js | Timed out

• A second run passed
• I see it mentioned in bugs bug 585361, bug 551746, bug 569736 and bug 593064 but not specifically filed for it.

XULRunner

No problems.

Source

No problems.

Build

No problems.

Signing

NOTE: use new signcode keys from d:/2011-keys and QUIT SIGNCODE when you're done with keymaster

• I had to fix to clean up the documentation (changes).
• Done

Firefox Signing

No problems.

L10N verify

No problems.

Updates

No problems.

Update verify

Failed due to bug 719436. That's been backed out and the RUNTIME tags moved to the fix. Need to rerun verify. (nthomas)

• Re-triggered the jobs

Major Update

Major update generation is not kicked off automatically. You should only create them once the final builds for the "latest" build are ready. For instance, the 3.6.26 builds got created a week before the 10.0 release but the "final" 10.0 builds were not created after few days.

Follow instructions from the Major update documentation.

XXX: TODO once 10.0 finals are signed and up

Reset reserved slaves

nthomas reset the value last night.

Run backupsnip

# cltbld@aus2-staging
-bash-3.2$ cd /opt/aus2/snippets/staging
-bash-3.2$ time ~/bin/backupsnip Firefox-3.6.26-build1-beta
real    151m44.062s
user    0m14.306s
sys     1m16.829s

Check permissions / AV scan

• XXX: It didn't trigger after updates were done; is that a bug?
• Triggered prior to push to beta on Thursday noon with:
  • Property 1 name: script_repo_revision -> FIREFOX_3_6_26_RELEASE
  • Property 2 name: release_config -> mozilla/release-firefox-mozilla-1.9.2.py

No problems.

Push to beta

 cd /opt/aus2/snippets/staging
 time ~/bin/pushsnip Firefox-3.6.26-build1-beta
 real    28m21.040s
 Running ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com touch /opt/aus2/incoming/3
 ---------------------------------------------------------------------------
 real    73m36.243s

Build 2

Preparing to start Automation

• Set clobbers for "any master" "mozilla-1.9.2" "any builder"
• Reserve slaves

echo "10" > master/reserved_slaves_bm08-build1 

• double-landed automation configs
• Tagged buildbot-configs, buildbotcustom, & tools with build & release tags FIREFOX_3_6_26_{BUILD2,RELEASE}
• update and reconfigure the master

make update && make checkconfig && make reconfig

• start automation

source ../bin/activate
cd /builds/buildbot/build1/master
# DRY RUN
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001
# REAL-REAL
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

already done for build 1

E-mail mirrors@mozilla.org

already done in build 1

Tag

No problems.

Bouncer Submitter

Already submitted in build 1

Source

No problems

Start autosign

NOTE: use new signcode keys from d:/2011-keys

Instructions are in CombinedSigning

Started.

Build

Firefox

No problems

Firefox repacks

Linux: done, no problems Win32: done, no problems Mac: done, no problems

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Updated build number and ran sign_xulrunner_3_6_26.sh on keymaster, no problems

Partner Repack

N/A

Major Update

Major update generation is not kicked off automatically. You should only create them once the final builds for the "latest" build are ready. For instance, the 3.6.26 builds got created a week before the 10.0 release but the "final" 10.0 builds were not created after few days.

Follow instructions from the Major update documentation.

Reset reserved slaves

QUIT SIGNCODE on keymaster and reset reserved slaves to 0 on bm08

Run backupsnip

NOTE: Remember to do this at least an hour ahead of the expected "go to beta" email.

Push files to internal mirrors

Done the day before release OR ASAP for chemspills: Mirrors Policy

Push to external mirrors

if this is a chemspill you will want to ping justdave (or other sysadmin) in #release-drivers 
and ask them to enable CDN

Update XULRunner wiki page

For major releases or chemspills, update the links on:

• https://developer.mozilla.org/en/XULRunner
• https://developer.mozilla.org/en/Gecko_SDK

Update symlinks

Once we're signed off on the release channel.

Remove index.html files

If you created them earlier to hide the release.