Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

Signed-off Revision(s)

L10N changesets

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110527093235	mv-moz2-linux-ix-slave05	56 mins, 50 secs
	Linux64		moz2-linux64-slave02	1 hrs, 5 mins, 46 secs
	Mac
	Windows		mw32-ix-slave14	2 hrs, 45 mins, 4 secs

Notes

Build 1

Setup

Marked "Any master", "mozilla-beta", "Any builder" for a clobber
Set reserved_slaves to 6
Hit "ship it" button on "Firefox 5 Beta Build 3" (https://l10n-stage-sj.mozilla.org/shipping/milestones)
Landed configs, merged to production
Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_5_0b3_RELEASE and FIREFOX_5_0b3_BUILD1
Updated pm03 checkouts & reconfiged
Ran release_sanity dry run with:

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 5.0b3 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py --dryrun localhost:9010

Gave this error, because the naming of releases on the dashboard changed bug 660319.

2011-05-27 08:44:41,862 : INFO : Comparing l10n changesets on dashboard https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx5.0b3 to on-disk l10n-changesets_mozilla-beta ...
2011-05-27 08:44:41,900 : ERROR : cannot find l10n dashboard at https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx5.0b3
2011-05-27 08:44:41,900 : ERROR : Error verifying l10n dashboard changesets

Did the following to manually verify the on-disk l10n changesets:

wget -O ~/l10n-changesets_mozilla-beta https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx5_beta_b3
diff ~/l10n-changesets_mozilla-beta l10n-changesets_mozilla-beta

To start the release (-b is to force the sendchange, even though a check has failed):

PYTHONPATH=.:tools/lib/python/ python tools/buildbot-helpers/release_sanity.py -u bhearsum -V 5.0b3 --branch mozilla-beta --build-number 1 -c release-firefox-mozilla-beta.py -b localhost:9010

Tag

After 10 minutes no linux slaves on pm03 were free, so I moved mv-moz2-linux-ix-slave10 from pm01 to pm03 with slavealloc. Once it connected, I emptied out the "Locked Master" field to allow it to be bounced around if slave alloc wishes.
No issues with tagging

Start Autosign

Started signcodepwd.exe
Ran the following in the cygwin terminal:

PRODUCT=firefox
VERSION=5.0b3
BUILD=1
TAG=FIREFOX_5_0b3_RELEASE
REPO='releases/mozilla-beta'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-beta.py

cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Bouncer Submitter

No problems.

Source

No problems.

Build

Linux: No problems
Linux64: No problems
Windows: No problems
Mac: No problems

Repack

Linux: No problems
Linux64: No problems
Windows: No problems
Mac: No problems

Signing

No problems

L10n Verify

"Only in..." lines for ak, bn-BD, es-AR, id, ja, ja-JP-mac, ku, vi
- Expected and ignorable because these are new locales in 5.0b3
A few string changes for existing locales

PASS.

Updates (5.0b[12] -> 5.0b3)

No problems

Update Verify

Configs weren't bumped properly due to two bugs: bug 660625 and bug 658083. Had to fix them manually (also this change).

After landing fixed configs and re-running the failed chunks, results are as follows:

Linux: PASS
Linux64: PASS
Mac: PASS
Windows: PASS

Reset reserved_slaves

production-master03:/builds/buildbot/builder_master

echo 0 > reserved_slaves_pm03

Updates (4.0 -> 5.0)

Manually created patcher config and update verify configs.

Generation was done on mv-moz2-linux-ix-slave13, using the UPDATE_PACKAGING_R14 tag, and re-using the updates directory from the 5.0b[12] -> 5.0b3 updates run, to save time & energy.

cd /builds/slave/rel-beta-updates/build/patcher-configs
cvs up

perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

cd temp/firefox
ln -s 5.0b3 4.0.1-5.0b3
cd ../..

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz20-branch-major-update-patcher2.cfg

# remove partials for 4.0.1
cd temp/firefox/4.0.1-5.0b3
find aus2{,.test} -name partial.txt -exec rm {} \;

# check releasetest == beta
bash /builds/slave/rel-beta-updates//tools/release/compare-channel-snippets.sh aus2 beta aus2.test releasetest

# push up to aus
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b3-build1-4.0.1-build1-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-5.0b3-build1-4.0.1-build1

# push test snippets live
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/backupsnip Firefox-5.0b3-build1-4.0.1-build1-test'
bash -c 'ssh -l cltbld -i ~/.ssh/cltbld_dsa aus2-staging.mozilla.org ~/bin/pushsnip Firefox-5.0b3-build1-4.0.1-build1-test'

Update Verify

To force all of the verify builders, used the following script (warning: not OS X compatible):

for p in linux linux64 macosx64 win32; do
  for i in `seq 1 10`; do
    curl "http://production-master01.build.mozilla.org:8010/builders/release-mozilla-2.0-${p}_major_update_verify_${i}%2F10/force" >/dev/null
  done
done

Complete failure the first time, because of bug 660667. Once that was fixed, retriggered builders.

Unlike 5.0b2, our update verify configs aren't trying to test partials or missing locales, so we don't have any spurious failure. However, we still have update channel diffs for all platforms/locales:

diff -r source/firefox/defaults/pref/channel-prefs.js target/firefox/defaults/pref/channel-prefs.js
1,2c1,2
< //@line 2 "/builds/slave/rel-2.0-lnx-bld/build/browser/app/profile/channel-prefs.js"
< pref("app.update.channel", "release");
---
> //@line 2 "/builds/slave/rel-beta-lnx-bld/build/browser/app/profile/channel-prefs.js"
> pref("app.update.channel", "beta");
WARN: non-binary files found in diff

As well as some locale specific differences:

en-ZA: answers.xml, creativecommons.xml
ro: creativecommons.xml

Which all look like legitimate search engine changes.

And platform specific differences:

Mac:

Only in source/Firefox.app: force_plist_reload

Which is OK because bug 386760 removed that file in 5.0.

Windows:

Only in source/bin: D3DCompiler_42.dll
Only in source/bin: d3dx9_42.dll

Which is OK because these files were replaced with _43 versions when bug 648804 landed.

Generate channel switching snippets

We'll use the 4.0.1 snippets for this, which has all of the locales we care about, plus a (harmless) few extra. Like 5.0b2, we put them in a bogus buildID of 30000000000000 to keep the channel switching snippets separate from more specific ones.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
mkdir -p Firefox-5.0b3-build1-channel-switching/Firefox/
rsync -a Firefox-5.0b3-build1-4.0.1-build1/Firefox/4.0.1 Firefox-5.0b3-build1-channel-switching/Firefox/
cd Firefox-5.0b3-build1-channel-switching/Firefox/
mv 4.0.1 5.0
for d in 5.0/*; do 
  mv -v $d/* $d/30000000000000
done

Virus scan & permission check

Both green. No problems.

A/V Notification

Mail sent successfully from automation.

Push Files

As in 5.0b2, we'll be pushing to a few internal mirrors rather than the whole network. To set this up, added the following to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

- firefox/releases/5.0b3

...and waited a few minutes, to ensure that the rsync servers have the file.

Then, forced the "release-mozilla-beta-push_to_mirrors" builder with the following properties:

   release_config: mozilla/release-firefox-mozilla-beta.py
   script_repo_revision: FIREFOX_5_0b3_RELEASE

Unlike 5.0b2, didn't get enough uptake for "ready for releasetest testing" or final verify to fire, so I used "force build" to start them.

Final verify

Noticed that all Windows requests got pointed at 3crowd.net rather than dm-download02. bug 646076 should've forced them onto dm-download02. Re-opened the bug to work on a fix. Had some 404 errors on the first Windows run, re-running resulted in all green. This step is a conditional PASS.

Push snippets

"Firefox-5.0b3-build1" and "Firefox-5.0b3-build1-channel-switching" both only have a "5.0" version in them, so we only need one backupsnip between them. "Firefox-5.0b3-build1-4.0.1-build1" has 4.0 versions, and needs its own. As cltbld@aus2-staging, did the following for backups:

cd /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-5.0b3-build1
~/bin/backupsnip Firefox-5.0b3-build1-4.0.1-build1

And for the pushes:

~/bin/pushsnip Firefox-5.0b3-build1
~/bin/pushsnip Firefox-5.0b3-build1-4.0.1-build1
~/bin/pushsnip Firefox-5.0b3-build1-channel-switching

Build #	Tag	Changeset
1	GECKO50_2011052708_RELBRANCH	23d449276096
1	FIREFOX_5_0b3_RELEASE FIREFOX_5_0b3_BUILD1	c93fe6829c74

Releases/Firefox 5.0b3/BuildNotes

Contents

Notes About Releasing

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Notes

Build 1

Setup

Tag

Start Autosign

Bouncer Submitter

Source

Build

Repack

Signing

L10n Verify

Updates (5.0b[12] -> 5.0b3)

Update Verify

Reset reserved_slaves

Updates (4.0 -> 5.0)

Update Verify

Generate channel switching snippets

Virus scan & permission check

A/V Notification

Push Files

Final verify

Push snippets

Navigation menu

Search