Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

bug 700013 - sign_android-0.8.sh needs to be smarter about disk space available
bug 674327 - bump the disk space requirement for signing
bug 690365 - updates failed at trying to upload to aus2.beta

Build Engineers

lsblakk - Tracking bug: bug 695722

Signed-off Revision(s)

Build 1: 04778346a3b0

L10N changesets

Instructions on how to get them

Firefox: l10n-changesets
Fennec: json

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20111104165243	mv-moz2-linux-ix-slave08	2 hrs, 27 mins
	Linux64		moz2-linux64-slave08	3 hrs, 52 mins
	Mac		moz2-darwin10-slave46	4 hrs, 30 mins
	Windows		mw32-ix-slave11	2 hrs, 53 mins

Fennec

Build #	Type	Build ID	Build machine	Time to build
1	android	20111104164606	mv-moz2-linux-ix-slave16	1 hr, 43 mins
	linux		mv-moz2-linux-ix-slave19	1 hr, 10 mins
	macosx		bm-xserve09	2 hrs, 14 mins
	win32		mw32-ix-slave15	1 hr, 30 mins

Notes

Build 1

Syncing repos

mozilla-beta -> mozilla-release was done by Christian Legnitto (as well as updating browser/locales/shipped_locales to eliminate non-release locales that were in beta releases mn,sw,csb)

Did l10n repo sync on linux-ix-slave02 as per instructions

Diff mozconfigs

Updated the mozconfig_whitelist PGO difference in release configs that are known and intentional diff

Preparing to start Automation

Set clobber for 'any master', 'mozilla-release', 'any builder'
reserved 10 slaves

echo "10" > master/reserved_slaves_bm07-build1

Landed automation configs & l10n updates
Tagged buildbot-configs (production), buildbotcustom (production-0.8), tools (default) with build & release tags

hg tag -f FIREFOX_8_0_BUILD1 FIREFOX_8_0_RELEASE FENNEC_8_0_BUILD1 FENNEC_8_0_RELEASE

release_sanity dry run

 source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk
 -V 8.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py
 --dryrun --products firefox,fennec localhost:9001

dry-run failed verify_l10n_dashboard because the url for the dashboard was https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx8 and the release_sanity looks for https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx8.0

start automation

 source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk
 -V 8.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py
 --products -b firefox,fennec localhost:9001

not reconfiguring other masters, it's EOD Pacific time on a Friday, sent an email to release@mozilla.com to notify for Monday morning
returned linux-ix-slave02 to the pool now that l10n repos are synched

E-mail Metrics

Only for beta1 and 1.9.2 releases

E-mail mirrors@mozilla.org

email sent, cc'd release@mozilla.com

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

# created in ~/tmp/sign_8.0.sh
PRODUCT=firefox
VERSION=8.0
BUILD=1
TAG=FIREFOX_8_0_RELEASE
REPO='releases/mozilla-beta'
EMAIL=release@mozilla.com
KEYDIR=d:/2011-keys
RELCONFIG=mozilla/release-firefox-mozilla-beta.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Started auto-sign

bash -xe ~/tmp/sign_8.0.sh 2>&1 | tee sign_8.0.log

Build

Firefox

No problems

Fennec

Android

No problems

Desktop Builds

No problems

Desktop repacks

No problems

Unittests / Talos

QUESTION: should I be looking at https://tbpl.mozilla.org/?tree=Mozilla-Release&rev=04778346a3b0 for this? The build sendchange for rev = d03b51a9b2bd did not trigger tests

Document any oranges (per platform) for unittests - if possible, try to confirm it's known/random
Make sure there's no red/failures that we need to have a dev look at

XULRunner

Source

No problems

Build

No problems

Signing

Copied sign_xulrunner-8.0b6.sh to sign_xulrunner-8.0.sh, altered version/build vars and ran the script

No problems

Repacks

Firefox

No problems

Fennec

No problems

Partner Repack

Firefox

Had to force builds on linux,linux64,mac while win32 was successfully triggered

Fennec

N/A for this release, no maemo

Firefox Signing

No problems, but would have been if I hadn't gone on to sign android and discovered there was no more disk space. Seems like we need at least 30 GB (?) free to do signing now and the signing script isn't clearing enough space, re-opened bug 674327

Android Signing

First attempt failed because of 0GB disk space available Cleaned up binaries from signed-work and retried, second time no problems filed bug 700013

Android Signature Verification

No problems

L10N verify

No problems

Updates

Hit bug 690365 - updates failed at trying to upload to aus2.beta

Update verify

Had to trigger manually:

# On my local machine, to kick off update verify
for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    curl "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F10/force" >/dev/null
    sleep 5
  done
done

Kicked {1,2}/10 on macosx64 again, failed network access to snippets Second attempt at 2/10 went gree, 1/10 failed again - still wasn't reaching the snippet url even though I could in browser, rebuilt again.

Major Update

N/A

Update verify

N/A

Socorro Product Data Updates

N/A

Reset reserved slaves

echo 0 > master/reserved_slaves_bm07-build1

Check permissions / AV scan

Forced build on both.

Push files to internal mirrors

updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to add

- firefox/releases/8.0

wait 10 minutes (at least, did this at 11:05 Sat Nov 5, PDT )

# in /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-8.0-build1 2>&1 | tee lsblakk_0.8.out

real    55m15.793s
user    0m9.349s
sys     0m43.953s

forced the release_push_to_mirrors with the following:

release_config: mozilla/release-firefox-mozilla-release.py
script_repo_revision: FIREFOX_8_0_RELEASE

Final verification

Forced builders, no problems

Push to beta

N/A

Push index file to mirrors

Added the following lines to the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

+ firefox
+ firefox/releases
+ firefox/releases/8.0
+ firefox/releases/8.0/*/
+ firefox/releases/8.0/*/*/
+ firefox/releases/8.0/*/*/*/
+ firefox/releases/8.0/**index.html
- firefox/releases/8.0/**

Waited 10 minutes to ensure the rsync server picks up the changes, then removed the previously added "- firefox/releases/8.0" line. After that, ran the following to lay out the index files:

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/8.0
wget --no-check-certificate -O index.html https://bugzilla.mozilla.org/attachment.cgi?id=540139
sed -i -e 's/5/8/g' index.html 
for dir in `find . -type d -mindepth 1`; do cp -pv index.html $dir/; done

Forced 'release-mozilla-release-ready_for_releasetest_testing' builder

Push to external mirrors

Removed the following lines from the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude:

+ firefox
+ firefox/releases
+ firefox/releases/8.0
+ firefox/releases/8.0/*/
+ firefox/releases/8.0/*/*/
+ firefox/releases/8.0/*/*/*/
+ firefox/releases/8.0/**index.html
- firefox/releases/8.0/**

~~Added - firefox/releases/7.0.1 line to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude~~ REVERTED - this can be done a few days after 8.0 ships, but not before the traffic on the 7.0.1 update files subsides.

Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to 8.0

Publish Fennec

Push Files

# as ffxbld@stage
vim push_fennec_release.sh
# update VERSION and BUILD
./push_fennec_release.sh

Contents of push_fennec_release.sh:

export VERSION=8.0
export BUILDNUM=1
export RD=/home/ftp/pub/mozilla.org/mobile/releases
export CD=/home/ftp/pub/mozilla.org/mobile/candidates

mkdir $RD/$VERSION
cd $RD/$VERSION

rsync -av --exclude=unsigned --exclude=*.txt --exclude=*crashreporter* \
 --exclude=*tests* --exclude=*unaligned* --exclude=*old* --exclude=*install* \
 --exclude=*logs* \
 $CD/$VERSION-candidates/build$BUILDNUM/ $RD/$VERSION/

for p in linux macosx win32; do
    locales=`cd $CD/$VERSION-candidates/build$BUILDNUM/$p/; ls -1 --hide=en-US`
    for i in $locales; do
        cp -a $CD/$VERSION-candidates/build$BUILDNUM/$p/$i/install/*.xpi \
          $RD/$VERSION/$p/$i
    done
done

cd $RD
rm -f latest
ln -s $VERSION latest
exit 0

Android Market

Uploaded apk and did adjustments to recent changes for all locales, uploaded new screenshot, and updated the description text for de and ja as per bug 697513

Push XULRunner to Mirrors

ssh -i ~/.ssh/xrbld_dsa xrbld@stage
rsync -n -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/8.0-candidates/build1/ /pub/mozilla.org/xulrunner/releases/8.0/
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/8.0-candidates/build1/ /pub/mozilla.org/xulrunner/releases/8.0/

Update XULRunner wiki page

Updated Gecko SDK wiki page

Push to Release Channel

# cltbld@aus2-staging 
~/bin/pushsnip Firefox-8.0-build1 2>&1 | tee lsblakk_8.0.out

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
ln -s 8.0 latest-8.0
rm latest && ln -s latest-8.0 latest

Remove index.html files

cd /pub/mozilla.org/firefox/releases/8.0
find . -name index.html -exec echo rm -v {} \;
find . -name index.html -exec rm -v {} \;

Update browser choice website

filed bug 700682

Build #	Branch, Tags	Changeset
1	MOBILE80_2011110415_RELBRANCH, FENNEC_8_0_RELEASE, FENNEC_8_0_BUILD1	c85cee638277
1	GECKO80_2011110416_RELBRANCH, FIREFOX_8_0_BUILD1 FIREFOX_8_0_RELEASE	67f0ce993976

Releases/Firefox 8.0/BuildNotes

Contents

Notes About Releasing

Bugs hit

Build Engineers

Signed-off Revision(s)

L10N changesets

Tags

Build data

Firefox

Fennec

Notes

Build 1

Syncing repos

Diff mozconfigs

Preparing to start Automation

E-mail Metrics

E-mail mirrors@mozilla.org

Tag

Bouncer Submitter

Source

Start autosign

Build

Firefox

Fennec

Android

Desktop Builds

Desktop repacks

Unittests / Talos

XULRunner

Source

Build

Signing

Repacks

Firefox

Fennec

Partner Repack

Firefox

Fennec

Firefox Signing

Android Signing

Android Signature Verification

L10N verify

Updates

Update verify

Major Update

Update verify

Socorro Product Data Updates

Reset reserved slaves

Check permissions / AV scan

Push files to internal mirrors

Final verification

Push to beta

Push index file to mirrors

Push to external mirrors

Publish Fennec

Push Files

Android Market

Push XULRunner to Mirrors

Update XULRunner wiki page

Push to Release Channel

Update symlinks

Remove index.html files

Update browser choice website

Navigation menu

Search