Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• bug 757911 - db outage caused 90 minute delay in start of tagging
• bug 758064 - win32 repacks fail due to badly formed command line - 2-1/2 hour delay in win32 repacks (no QA impact per email)
• bug 758216 - hg.m.o 500 error during clone steps script not retried. rebuilt step.

Build Engineers

{hwine} - Tracking bug: bug 744594

Signed-off Revision(s)

Build 1: 491fd5c11f42

L10N changesets

Instructions on how to get them

• Firefox: https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx13_beta_b5
• Fennec: not building

Tags

Manually tag the automation code, then record the generated tags below. (details)

Build #	Branch, Tags	Changeset
1	GECKO130_2012052311_RELBRANCH FIREFOX_13_0b5_BUILD1 FIREFOX_13_0b5_RELEASE	5de6e4bd8ede

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20120523114940	linux-ix-slave28	1 hrs, 3 mins, 40 secs
	Linux64		linux64-ix-slave21	1 hrs, 4 mins, 13 secs
	Mac		moz2-darwin10-slave46	3 hrs, 17 mins, 47 secs
	Windows		w32-ix-slave37	3 hrs, 41 mins, 36 secs

Notes

Build 1

Diff mozconfigs

If this is a *.0 beta -> release. May also need to update the whitelist of nightly vs release differences for release sanity.

Preparing to start Automation

detailed instructions

• Set clobbers for the appropriate masters. Doing this 24-48 hours in advance should speedup the build.
  • clobberer

done

• Reserve slaves (you need ~5 each of linux, linux64, win32 (w32), and macosx64 (moz2-darwin10)). Doing this 2-3 hours in advance should speed things up. e.g.:

[cltbld@buildbot-master13 master]$ pwd
/builds/buildbot/build1/master
[cltbld@buildbot-master13 master]$ cat reserved_slaves
5

done

• For Firefox build1 only, create a symlink for your $version (eg 11.0b4, 10.0.2esr, 10.0.2, 3.6.27) (until bug 725839 is resolved)

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/
mkdir ../candidates/$version-candidates
ln -s ../candidates/$version-candidates $version-candidates

done

• Update l10n changesets for desktop, mobile
  • Instructions on how to get them

done

• • dashboard
• Land automation configs (are you or buildduty going to reconfig? great, merge. otherwise double-land only your configs)

done (double-landed)

• Tag buildbot-configs (production), buildbotcustom (production-0.8), & tools (default) with build & release tags. (details)
  • Sample command for Firefox 10.0 beta 5 build 1:

hg tag -f FIREFOX_10_0b5_RELEASE FIREFOX_10_0b5_BUILD1
hg tag -f FENNEC_10_0b5_RELEASE FENNEC_10_0b5_BUILD1
# bash shortcut: hg tag -f {FIREFOX,FENNEC}_10_0b5_{RELEASE,BUILD1}

• update and reconfigure the master

 ./manage_masters.py -H bm13-build1 -f production-masters.json -l
 ./manage_masters.py -H bm13-build1 -f production-masters.json check checkconfig show_revisions
 ./manage_masters.py -H bm13-build1 -f production-masters.json update checkconfig show_revisions
 ./manage_masters.py -H bm13-build1 -f production-masters.json reconfig show_revisions

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg -V 13.0b1 --branch mozilla-beta --build-number 1 \
    --release-config release-firefox-mozilla-beta.py --release-config release-fennec-mozilla-beta.py --products firefox,fennec  \
    --dryrun localhost:9001

• start automation ← monitor progress on buildbot (e.g. beta)
  • the same command as above but without the "--dryrun"
• reconfigure other masters (or ask the buildduty person to do this) if you have done a merge (and not double-land) to production

E-mail Metrics

Use the address "metrics-alerts < AT > mozilla < PERIOD > org". Note for first-time-releasers: your email will get held for moderator approval - that is expected. If it happens more than once, get help on #metrics.

For 13.0b5 it would be

Firefox,firefox,firefox,13.0b5,13.0

done

Tag

(90 minute delay due to bug 757911 - r-d@ notified via email) done

Bouncer Submitter

done

Source

done

Build

Firefox

Firefox repacks

Win32 failed due to bug 758064.

• aki double landed a fix: default production and tagged
• hwine needs to update+reconfig bm13, then rebuild repacks

Unittests / Talos

details

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

done

Build

done

L10N verify

done

Checksums

done

Updates

done

Update verify

done

Reset reserved slaves

done

Check permissions / AV scan

done

Push to internal mirrors

done

Final verification

Record receipt of "completed_final_verification" emails (1 per platform):

• Subject: [release] Firefox 13.0b5 build1: completed final_verification on macosx64
• Subject: [release] Firefox 13.0b5 build1: completed final_verification on linux
• Subject: [release] Firefox 13.0b5 build1: completed final_verification on win32

Note: final_verification on linux64 succeeded based on manual inspection, as no email sent. (500 error on download of one URL caused automation to fail. Manual inspection of that URL showed it was available.)

done

Going to Beta

Run backupsnip

NOTE: Remember to do this at least 2 hours ahead of the expected "go to beta" email. details

Remember to ssh to aus3-staging using your short LDAP name and then run sudo su - ffxbld once on. If you intend to use screen, initiate your screen session while you are still yourself.

done in 9 minutes

Push to beta

Wait for "go" from release managers
For Firefox:

• Publish (and record run time here): details
  • pushsnips took 6m16.831s
• on completion, email r-d that VERSION updates are live on the beta channel

done

Wait for confirmation from QA that each of the above releases is okay, then close the tracking ticket.