Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

I landed my changes on default and forgot to merge to production before tagging the repos
- release_sanity saved the day and I fixed it before starting anything
bug 720743 - Linux repack 6/6 timed out
- I hit "rebuild" and documented it
bug 720761 - The bump of the version on default for a release triggers dependent builds
- This does not affect the release but could suck several builders before the release jobs (build + tests happen on build machines for 3.6)
Should we have the mirrors email lower in the build notes? I'm a week ahead from the release day but the email template says "hey, we're shipping in 24hours".
- I guess just saying "24 hours" or "on this day" could make it unambiguous
- could we automate this as well?
bug 719436 update verify required some code back out - nthomas caught it
I had to clean up the XulRunner signing documentation (see the changes)
bug 722182 - investigate double-triggering of builders by downstream
Anti-virus didn't trigger after updates were done; is that a bug?
bug 722790 - mac update verify failed because we bumped the wrong mac platform
- fix submitted

Build Engineers

armenzg - Tracking bug: bug 719214

Signed-off Revision(s)

Build 1: 173bc943fe0d

L10N changesets

Instructions on how to get them

Firefox: [1]
Fennec: N/A

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120124074252	mv-moz2-linux-ix-slave04	40 mins, 5 secs
	Mac		bm-xserve11	2 hrs, 6 mins, 54 secs
	Windows		mw32-ix-slave03	1 hrs, 44 mins, 34 secs

Notes

Build 1

Preparing to start Automation

Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
- "any master", "1.9.2", "any builder"
Reserve slaves

echo 10 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

Update l10n changesets for desktop, mobile
Land automation configs
Tag buildbot-configs, buildbotcustom, & tools with build & release tags
- FIREFOX_3_6_26_{RELEASE,BUILD1}
update and reconfigure the master
- cd /builds/buildbot/build1
- make update && make reconfigure
start automation

 cd /builds/buildbot/build1/master
 source ../bin/activate
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001

The real thing:

cd /builds/buildbot/build1/master
 source ../bin/activate
 PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

Sent to "metrics-alerts < AT > mozilla < PERIOD > org"

Firefox,firefox,firefox,3.6.26,3.6
Firefox,firefox,firefox,3.6.27pre,3.6

E-mail mirrors@mozilla.org

Sent by lsblakk.

Tag

No problems.

Bouncer Submitter

No problems.

Source

No problems.

Start autosign

NOTE: use new signcode keys from d:/2011-keys

Instructions are in CombinedSigning

Build

Firefox

No problems.

Firefox repacks

Hit bug 720743 where one of the Linux repacks timed out when cloning mozilla-1.9.2
- I hit "rebuild" and carried on

Unittests / Talos

Look to see that they ran
Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
Make sure there's no red/failures that we need to have a dev look at

From release-mozilla-1.9.2-linux-opt-unittest-xpcshell-build9.txt.gz:

TEST-UNEXPECTED-FAIL | /builds/slave/test/build/xpcshell/tests/xpcom/unit/test_nsIProcess.js | test failed (with xpcshell return code: 0), see following log:

A second run passed
This is an intermittent orange (see bug 493778)

From release-mozilla-1.9.2-win32-opt-unittest-mochitest-other-build10.txt.gz:

TEST-UNEXPECTED-FAIL | chrome://mochikit/content/browser/browser/components/places/tests/perf/browser_ui_history_sidebar.js | Timed out

A second run passed
I see it mentioned in bugs bug 585361, bug 551746, bug 569736 and bug 593064 but not specifically filed for it.

XULRunner

No problems.

Source

No problems.

Build

No problems.

Signing

NOTE: use new signcode keys from d:/2011-keys and QUIT SIGNCODE when you're done with keymaster

I had to fix to clean up the documentation (changes).
Done

Firefox Signing

No problems.

L10N verify

No problems.

Updates

No problems.

Update verify

Failed due to bug 719436. That's been backed out and the RUNTIME tags moved to the fix. Need to rerun verify. (nthomas)

Re-triggered the jobs

Reset reserved slaves

nthomas reset the value last night.

Run backupsnip

# cltbld@aus2-staging
-bash-3.2$ cd /opt/aus2/snippets/staging
-bash-3.2$ time ~/bin/backupsnip Firefox-3.6.26-build1-beta
real    151m44.062s
user    0m14.306s
sys     1m16.829s

Check permissions / AV scan

Triggered prior to push to beta on Thursday noon with:
- Property 1 name: script_repo_revision -> FIREFOX_3_6_26_RELEASE
- Property 2 name: release_config -> mozilla/release-firefox-mozilla-1.9.2.py

No problems.

Push to beta

 cd /opt/aus2/snippets/staging
 time ~/bin/pushsnip Firefox-3.6.26-build1-beta
 real    28m21.040s
 Running ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com touch /opt/aus2/incoming/3
 ---------------------------------------------------------------------------
 real    73m36.243s

Build 2

Preparing to start Automation

Set clobbers for "any master" "mozilla-1.9.2" "any builder"
Reserve slaves

echo "10" > master/reserved_slaves_bm08-build1

double-landed automation configs
Tagged buildbot-configs, buildbotcustom, & tools with build & release tags FIREFOX_3_6_26_{BUILD2,RELEASE}
update and reconfigure the master

make update && make checkconfig && make reconfig

start automation

source ../bin/activate
cd /builds/buildbot/build1/master
# DRY RUN
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001
# REAL-REAL
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -p firefox -V 3.6.26 --branch  mozilla-1.9.2 --build-number 2 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

already done for build 1

E-mail mirrors@mozilla.org

already done in build 1

Tag

No problems.

Bouncer Submitter

Already submitted in build 1. Someone forced the builder using the buildbot webui though. It failed with '400 Bad request' errors, and the products/locations like fine in bouncer.

Source

No problems

Start autosign

NOTE: use new signcode keys from d:/2011-keys

Instructions are in CombinedSigning

Started.

Build

Firefox

No problems

Firefox repacks

Linux: done, no problems Win32: done, no problems Mac: done, no problems

Unittests / Talos

Look to see that they ran
Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Updated build number and ran sign_xulrunner_3_6_26.sh on keymaster, no problems

Partner Repack

N/A

Firefox Signing

No problems

L10N verify

No problems

Updates

2 runs of updates were triggered by 'downstream' - one bailed on commit_patcher_config because the other had already done the version bump from 3.6.25 to 3.6.26 build2.

The create_buildN_snippets step was red due to bug 722203, which is fixed below.

backupsnip failed after all 5 attempts to run it with retry failed after taking more than 2 hours. Recover manually

# aus2-staging
~/bin/backupsnip Firefox-3.6.26-build2-test

# mv-moz2-linux-ix-slave04
cd /builds/slave/rel-m-192-updates/build/temp/firefox/3.6.25-3.6.26
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.26-build2
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.26-build2-beta
bash /builds/slave/rel-m-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

The last produces lots of this

WARN: aus2/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/fr/release/partial.txt exists
  but aus2.test/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/fr/releasetest/partial.txt doesn't
WARN: aus2/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/fr/release/complete.txt exists
  but aus2.test/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/fr/releasetest/complete.txt doesn't
WARN: aus2/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/de/release/partial.txt exists
  but aus2.test/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/de/releasetest/partial.txt doesn't
WARN: aus2/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/de/release/complete.txt exists
  but aus2.test/Firefox/3.6.26/WINNT_x86-msvc/20120124074252/de/releasetest/complete.txt doesn't

which is likely fallout from bug 722203.

Update verify

Had to force these with:

for p in linux macosx win32; do
    for i in 1 2 3 4 5 6 7 8 9 10; do
      echo $p $i
      curl -s "http://buildbot-master08.build.sjc1.mozilla.com:8001/builders/release-mozilla-1.9.2-${p}_update_verify_${i}%2F10/force" >/dev/null
      sleep 5
    done
done

which failed because the pushsnip hadn't been run (what a noobie move). Anyway, take this chance to fix up the buildN snippets before we actually run it

# mv-moz2-linux-ix-slave04
cd /builds/slave/rel-m-192-updates/build/temp/firefox/3.6.25-3.6.26
for d in aus*; do 
  rsync -a $d/ $d.bak/
done
rm -rf aus2*/Firefox/3.6.26
export PYTHONPATH=/builds/slave/rel-m-192-updates/tools/lib/python:
pushd /builds/slave/rel-m-192-updates/tools/
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=592556 | patch -p1
popd
python /builds/slave/rel-m-192-updates/tools/release/generate-candidate-build-updates.py --brand Firefox --product firefox --app-name browser --version 3.6.26 --app-version 3.6.26 --old-version 3.6.25 --old-app-version 3.6.25 --build-number 2 --old-build-number 1 --channel betatest --channel releasetest --channel beta --stage-server stage.mozilla.org --old-base-snippet-dir . --workdir . --hg-server http://hg.mozilla.org/ --source-repo releases/mozilla-1.9.2 --verbose --platform win32 --platform macosx --platform linux --channel release --generate-partials | tee buildN.log
bash /builds/slave/rel-m-192-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

Still get errors about 3.6b1/b2/b3/b4, but this appears to be long-standing issue and I'm not going to block on it.

rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/Firefox/3.6.26 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.26-build2-test/Firefox/
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/Firefox/3.6.26 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.26-build2/Firefox/
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.beta/Firefox/3.6.26 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.26-build2-beta/Firefox/

# aus2-staging
~/bin/pushsnip Firefox-3.6.26-build2-test

then redo the update verify forcing. All green at last.

Major Update

Major update generation is not kicked off automatically. You should only create them once the final builds for the "latest" build are ready. For instance, the 3.6.26 builds got created a week before the 10.0 release but the "final" 10.0 builds were not created after few days.

Follow instructions from the Major update documentation.

Landed change
Reconfigure bm08
"Force build" for the major update builder

XXX: Waiting to push on 1/2/2012

Update verify

bug 722790 - We bumped mac to the wrong platform
- Fix landed
- I had to tag with the FIREFOX10 runtime tag - bug 722860
- re-triggered
- all good

Reset reserved slaves

QUIT SIGNCODE on keymaster and reset reserved slaves to 0 on bm08

Run backupsnip

NOTE: Remember to do this at least an hour ahead of the expected "go to beta" email.

# cltbld@aus2-staging
-bash-3.2$ screen -R armenzg
-bash-3.2$ cd /opt/aus2/snippets/staging
-bash-3.2$ time ~/bin/backupsnip Firefox-3.6.26-build2-beta
real    143m33.360s
user    0m14.508s
sys     1m15.349s

Push to beta

~/bin/pushsnip Firefox-3.6.26-build2-beta

Check permissions / AV scan

Triggered before going to beta:

Property 1 name: script_repo_revision -> FIREFOX_3_6_26_RELEASE
Property 2 name: release_config -> mozilla/release-firefox-mozilla-1.9.2.py

Done. No problems.

Push files to internal mirrors

There was no email to push to internals but a late email to push to external mirrors. See push to external mirrors.

Push to external mirrors

Force build the builder @ 6:23AM
Updated "rsyncd-mozilla-current.exclude" to point to 3.6.26

Final verification

Push XULRunner to Mirrors

rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \
/pub/mozilla.org/xulrunner/nightly/3.6.26-candidates/build2/ \
/pub/mozilla.org/xulrunner/releases/3.6.26/

Update XULRunner wiki page

For major releases or chemspills, update the links on:

Push to Release Channel

# cltbld@aus2-staging
screen -R armenzg
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip Firefox-3.6.26-build2
real    199m53.372s
user    0m14.808s
sys     1m17.638s

Pushed with:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/pushsnip Firefox-3.6.26-build2
# for phx it finished in 30m32.554s

Update symlinks

Once we're signed off on the release channel.

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-3.6 && ln -s 3.6.26 latest-3.6

Push MU

After email from release-coordinator, ran:

cd /opt/aus2/snippets/staging/
~/bin/backupsnip Firefox-3.6.26-build2-10.0-build1-MU
~/bin/pushsnip Firefox-3.6.26-build2-10.0-build1-MU

Build #	Branch, Tags	Changeset

	GECKO19226_2012012406_RELBRANCH FIREFOX_3_6_26_BUILD1 FIREFOX_3_6_26_RELEASE	982f9c134751

Releases/Firefox 3.6.26/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Notes

Build 1

Preparing to start Automation

E-mail Metrics

E-mail mirrors@mozilla.org

Tag

Bouncer Submitter

Source

Start autosign

Build

Firefox

Firefox repacks

Unittests / Talos

XULRunner

Source

Build

Signing

Firefox Signing

L10N verify

Updates

Update verify

Reset reserved slaves

Run backupsnip

Check permissions / AV scan

Push to beta

Build 2

Preparing to start Automation

E-mail Metrics

E-mail mirrors@mozilla.org

Tag

Bouncer Submitter

Source

Start autosign

Build

Firefox

Firefox repacks

Unittests / Talos

XULRunner

Source

Build

Signing

Partner Repack

Firefox Signing

L10N verify

Updates

Update verify

Major Update

Update verify

Reset reserved slaves

Run backupsnip

Push to beta

Check permissions / AV scan

Push files to internal mirrors

Push to external mirrors

Final verification

Push XULRunner to Mirrors

Update XULRunner wiki page

Push to Release Channel

Update symlinks

Push MU

Navigation menu

Search