Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

• NEW - l10n webapp had beta locales in 8.0.1 milestone, worked around it, Axel resolved without a bug
• NEW - bug 702631 - partner repacks only triggered on win32 (build1)
• KNOWN - bug 690365 - failed to upload aus2.beta/ snippets
• KNOWN (but unfiled) - bug 704028 - timeout creating bundle for xulrunner
• NEWish - Problems cleaning up 8.0.1 build1 files on internal mirrors & CDN, bug 703729
• KNOWN - bug 664539 AUS returns empty response occasionally

Build Engineers

nthomas - Tracking bug: bug 701537

Signed-off Revision(s)

Build 1: 6e453b4f7056
Build 2: 463b6cbdef7c

L10N changesets

Instructions on how to get them

• None for chemspill.

Tags

Build #	Branch, Tags	Changeset
1	GECKO80_2011110416_RELBRANCH, FIREFOX_8_0_1_BUILD1	8877f7af6bd5
2	GECKO80_2011110416_RELBRANCH, FIREFOX_8_0_1_BUILD2 FIREFOX_8_0_1_RELEASE	463b6cbdef7c

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20111115014305	mv-moz2-linux-ix-slave08	2 hrs, 25 mins, 37 secs
	Linux64		moz2-linux64-slave03	3 hrs, 44 mins, 53 secs
	Mac		moz2-darwin10-slave29	4 hrs, 37 mins, 58 secs
	Windows		mw32-ix-slave13	3 hrs, 4 mins, 34 secs
2	Linux	20111120135848	mv-moz2-linux-ix-slave08	2 hrs, 27 mins, 45 secs
	Linux64		moz2-linux64-slave03	3 hrs, 32 mins, 18 secs
	Mac		moz2-darwin10-slave29	4 hrs, 37 mins, 45 secs
	Windows		mw32-ix-slave13	3 hrs, 8 mins, 11 secs

Fennec

N/A

Notes

This is a desktop-only release.

Build 1

Syncing repos

n/a

Diff mozconfigs

n/a

Preparing to start Automation

• Set clobber for 'any master', 'mozilla-release', 'any builder'
• reserved 10 slaves

echo "10" > master/reserved_slaves_bm07-build1

• l10n changesets: didn't hit Ship on 8.0.1 milestone, it includes csb, mn, and sw locales. There are no changes on there otherwise, and we wouldn't take them anyway. Advised Axel
• Land automation configs on default and production (ie did not merge from default, too much pending)
• Tag buildbot-configs, buildbotcustom, tools with build & release tags
• update and reconfigure the master
• release_sanity dry run

source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
  -V 8.0.1 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
  --dryrun --products firefox localhost:9001
2011-11-15 01:24:46,790 : INFO : Comparing l10n changesets on dashboard https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx8.0.1 to on-disk l10n-changesets_mozilla-release ...
2011-11-15 01:24:47,047 : ERROR : 	locale mn missing in config
2011-11-15 01:24:47,047 : ERROR : 	locale csb missing in config
2011-11-15 01:24:47,048 : ERROR : 	locale sw missing in config
2011-11-15 01:24:47,048 : ERROR : Error verifying l10n dashboard changesets

which is the goof in the l10n system, so I'm overriding it

• start automation

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
 -V 8.0.1 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
 --products firefox localhost:9001 -b

E-mail Metrics

Sent

Firefox,firefox,firefox,8.0.1,8.0

E-mail mirrors@mozilla.org

email sent, cc'd release@mozilla.com

Tag

no problems

Bouncer Submitter

Vanilla and EU Ballot jobs had no problems

Source

No problems

Start autosign

Made sure 40G was free, then

# created in ~/tmp/sign_8.0.1.sh
PRODUCT=firefox
VERSION=8.0.1
BUILD=1
TAG=FIREFOX_8_0_1_RELEASE
REPO='releases/mozilla-release'
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=mozilla/release-firefox-mozilla-release.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

No problems

Fennec

N/A

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Copied sign_xulrunner-8.0.sh to sign_xulrunner-8.0.1.sh, altered version/build vars and ran the script. No problems

Repacks

Firefox

No problems

Fennec

N/A

Partner Repack

Firefox

Partner repacks have been triggered for win32 platform only. Filed bug 702631. Triggered linux, linux64 and mac partner repacks manually.

Firefox Signing

No problems

Android Signing

N/A

L10N verify

No problems

Updates

The Plan (TM):

1a) users on 4.0 - 7.0.1 should get updates to 8.0.1
1b) users who want to download Firefox should find 8.0.1 on mozilla.org
2a) users on 8.0 mac should get idletime/manual check-for-updates to 8.0.1
2b) users on 8.0 win32/linux should not get idle-time updates to 8.0.1.

To achieve this we'll need to remove Firefox/8.0/{Linux*,WINNT*}/20111104165243 from the snippets generated.

Automation failed at uploading non-existent aus2.beta dir - bug 690365. Ran releasetest==release check manually

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/temp/firefox/8.0-8.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

PASS.

Fix up 8.0 -> 8.0.1 platforms

# aus2-staging
cd /opt/aus2/incoming/3/Firefox/8.0
rm -rfv {Linux_x86*,WINNT*}/20111104165243

cd /opt/aus2/snippets/staging
rsync -a Firefox-8.0.1-build1/ Firefox-8.0.1-build1-limited_8.0_8.0.1/
rm -rfv Firefox-8.0.1-build1-limited_8.0_8.0.1/Firefox/8.0/{Linux_x86*,WINNT*}/20111104165243
rsync -a Firefox-8.0.1-build1-test/ Firefox-8.0.1-build1-limited_8.0_8.0.1-test/
rm -rfv Firefox-8.0.1-build1-limited_8.0_8.0.1-test/Firefox/8.0/{Linux_x86*,WINNT*}/20111104165243

Did the first bit of cleanup in PHX too.

7.0.1 -> 8.0.1 partials

7.0.1 -> 8.0.1 partials are nice to have too since we still have a sizable number of users on 7.0.1:

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/
curl -sL "https://bugzilla.mozilla.org/attachment.cgi?id=574762" > patcher-configs/mozRelease-special.cfg
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-special.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-special.cfg

cd /builds/slave/rel-m-rel-updates/build/temp/firefox/7.0.1-8.0.1/ftp/firefox/nightly/8.0.1-candidates/build1
find . -type f -exec chmod 644 '{}' ';'
find . -type d -exec chmod 755 '{}' ';'
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/8.0.1-candidates/build1/

cd /builds/slave/rel-m-rel-updates/build/temp/firefox/7.0.1-8.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-8.0.1-build1-7.0.1partials-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-8.0.1-build1-7.0.1partials

# aus2-staging
~/bin/backupsnip Firefox-8.0.1-build1-7.0.1partials-test
~/bin/pushsnip   Firefox-8.0.1-build1-7.0.1partials-test

Added the Firefox-8.0.1-Partial-7.0.1 product and locations to bouncer.

Update verify

Landed a config change to get the right testing matrix, and updated the RUNTIME tag.

Forcing the jobs:

# On my local machine, to kick off update verify
for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    curl "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F10/force" >/dev/null
    sleep 5
  done
done

Major Update

N/A for now

Update verify

Socorro Product Data Updates

N/A

Reset reserved slaves

echo 0 > master/reserved_slaves_bm07-build1

Check permissions / AV scan

Forced. Made sure that clamd and extract_and_run_command.py were both niced off on stage so that it didn't interfere with uploads/rsyncs etc.

Got sick of waiting for the AV scan to process win32 dead last, after other platforms and unsigned, so did

# ffxbld@stage 
python /usr/local/bin/extract_and_run_command.py -j2 clamdscan -m --no-summary -- /pub/mozilla.org/firefox/nightly/8.0.1-candidates/build1/win32* 2>&1 | tee /tmp/win32-scan.log

Finally finishes AV after 4 hours.

Push files to internal mirrors

Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to add

- firefox/releases/8.0.1

Waited at least 10 minutes for that to propagate.

Forced the release_push_to_mirrors with the following:

release_config: mozilla/release-firefox-mozilla-release.py
script_repo_revision: FIREFOX_8_0_1_RELEASE

Final verification

Forced, no problems

Respin

Issues with the Roboform fix mean we'll need build2. Deleted firefox/releases/8.0.1 to get a start on cleaning the internal mirrors up (they only remove 10 files+dirs per sync). Also filed bug 703729 to expedite that when I also needed to cleanup some old betas in prerelease too.

Build 2

Syncing repos

n/a

Diff mozconfigs

n/a

Preparing to start Automation

• Set clobber for 'any master', 'mozilla-release', 'any builder'
• reserved 10 slaves

echo "10" > master/reserved_slaves_bm07-build1

• no l10n changesets to deal with
• Land automation configs on default and production (ie did not merge from default, too much pending)
• Tag buildbot-configs, buildbotcustom, tools with build & release tags
• update and reconfigure the master
• release_sanity dry run

source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
 -V 8.0.1 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
 --dryrun --products firefox localhost:9001

• No errors so Axel must have fixed the l10n webapp. Start automation

PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u nthomas \
 -V 8.0.1 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
 --products firefox localhost:9001

E-mail Metrics

n/a for build2

E-mail mirrors@mozilla.org

Sent an updated ETA for release.

Tag

no problems

Bouncer Submitter

n/a for build > 1

Source

no problems

Start autosign

Updated the BUILD definition in ~/tmp/sign_8.0.1.sh

Build

Firefox

No problems

Fennec

n/a

Unittests / Talos

1. Look to see that they ran
2. Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
3. Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

Timed out creating a bundle, filed bug 704028. Forced a new job, which worked.

Build

no problems

Signing

Updated ~/sign_xulrunner-8.0.1.sh

Repacks

Firefox

no problems

Fennec

n/a

Partner Repack

Firefox

no problems, the fix in bug 702631 worked fine

Firefox Signing

no problems

Android Signing

n/a

L10N verify

no problems

Updates

The Plan (TM, same as build 1):

1a) users on 4.0 - 7.0.1 should get updates to 8.0.1
1b) users who want to download Firefox should find 8.0.1 on mozilla.org
2a) users on 8.0 mac should get idletime/manual check-for-updates to 8.0.1
2b) users on 8.0 win32/linux should not get idle-time updates to 8.0.1.

To achieve this we'll need to remove Firefox/8.0/{Linux*,WINNT*}/20111104165243 from the snippets generated.

Automation failed at uploading non-existent aus2.beta dir - bug 690365. Ran the releasetest==release check manually

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/temp/firefox/8.0-8.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest

PASS.

Fix up 8.0 -> 8.0.1 platforms

Before pushsnip ran (ie while backupsnip was running) was able to fix up the test snippets

# aus-staging
cd /opt/aus2/snippets/staging/Firefox-8.0.1-build2-test/Firefox/8.0
rm -rfv {Linux_x86*,WINNT*}/20111104165243

and the release snippets prior to upload

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/temp/firefox/8.0-8.0.1/aus2/Firefox/8.0
rm -rfv {Linux_x86*,WINNT*}/20111104165243

7.0.1 -> 8.0.1 partials

Backed out the previous update verify fix (038011808b9d) before updates ran.

# cltbld@mv-moz2-linux-ix-slave17
cd /builds/slave/rel-m-rel-updates/build/
curl -sL "https://bugzilla.mozilla.org/attachment.cgi?id=575812" > patcher-configs/mozRelease-special.cfg
perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-special.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-special.cfg

cd /builds/slave/rel-m-rel-updates/build/temp/firefox/7.0.1-8.0.1/ftp/firefox/nightly/8.0.1-candidates/build2
find . -type f -exec chmod 644 '{}' ';'
find . -type d -exec chmod 755 '{}' ';'
rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/8.0.1-candidates/build2/

cd /builds/slave/rel-m-rel-updates/build/temp/firefox/7.0.1-8.0.1
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh aus2 release aus2.test releasetest
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-8.0.1-build2-7.0.1partials-test
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-8.0.1-build2-7.0.1partials

# aus2-staging
~/bin/backupsnip Firefox-8.0.1-build2-7.0.1partials-test
~/bin/pushsnip   Firefox-8.0.1-build2-7.0.1partials-test

Update verify

Landed a config change to get the right testing matrix, and updated the RUNTIME tag.

Forcing the jobs:

# On my local machine, to kick off update verify
for p in linux linux64 macosx64 win32; do
 for i in 1 2 3 4 5 6 7 8 9 10; do
   curl -s "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F10/force" >/dev/null
   sleep 5
 done
done

Linux32/Linux64/Windows no problems. Mac 1/10 failed with some sort of download error for a dmg, ok on rebuild.

Major Update

Updated the 1.9.2 release config and forced the major update builder.

Update verify

All green, but in the logs we're not removing these 3.6.24 files:

• all platforms: modules/SpatialNavigation.js
• linux only: icons/document.png

Those are removed by bug 670880 and bug 504645 respectively, without modifying removed-files.in correctly. bug 705974 filed to fix that up in the next MU.

Socorro Product Data Updates

n/a

Reset reserved slaves

echo 0 > master/reserved_slaves_bm07-build1

Check permissions / AV scan

Both forced. Made sure that clamd and extract_and_run_command.py were both niced off on stage so that it didn't interfere with uploads/rsyncs etc.

No problems with either.

Push files to internal mirrors

Need to make sure download-cdnetworks.mozilla.net has purged the build1 files:

# stage
cd /pub/mozilla.org/firefox/nightly/8.0.1-candidates/build2/
find linux-* mac/ win32* update/ -type f | grep -vE 'asc|checksums|xpi|zip|txt|log' | sort > /tmp/cdnetworks-list.log
(while read line; do
  url="http://download-cdnetworks.mozilla.net/pub/mozilla.org/firefox/releases/8.0.1/$line"
  echo "$url"
  curl -sI "$url"
  echo
done < "/tmp/cdnetworks-list.log") | tee /tmp/cdnetworks-check2.log

There are 388 responses of 200 instead 404. Reopened bug 703729 and IT purged the files from the CDN. Ended up disabling the CDN because it was still serving build1 files, and waited for the mirrors.

Final verification

Some empty updates bug 664539, otherwise OK.

Push to beta

n/a

Push index file to mirrors

n/a

Push to external mirrors

• Removed the following line from stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

- firefox/releases/8.0.1

• Forced the builder with script_repo_revision property set to FIREFOX_8_0_1_RELEASE
• Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to point at 8.0.1 (and 3.6.24)

Publish Fennec

n/a

Push Files

Android Market

Push XULRunner to Mirrors

# xrbld@stage
rsync -n -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \
 --exclude=*.checksums --exclude=jsshell* \
 /pub/mozilla.org/xulrunner/nightly/8.0.1-candidates/build2/ \
 /pub/mozilla.org/xulrunner/releases/8.0.1/
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \
 --exclude=*.checksums --exclude=jsshell* \
 /pub/mozilla.org/xulrunner/nightly/8.0.1-candidates/build2/ \
 /pub/mozilla.org/xulrunner/releases/8.0.1/

Update XULRunner wiki page

Updated Gecko SDK wiki page

Push to Release Channel

~/bin/backupsnip Firefox-8.0.1-build2
# real    74m39.435s
~/bin/backupsnip Firefox-8.0.1-build2-7.0.1partials
# real    1m32.767s

~/bin/pushsnip Firefox-8.0.1-build2-7.0.1partials
~/bin/pushsnip Firefox-8.0.1-build2

Landed unthrottling patch, tagged for production, and filed bug 704424.

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-8.0 && ln -s 8.0.1 latest-8.0

Update browser choice website

Filed bug 704359