Releases/Firefox 7.0/BuildNotes

From MozillaWiki
Jump to: navigation, search

Contents

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

  • bug 688669 - don't send the "all signed builds available" for Fennec until after upload is successful
  • bug 688764 xulrunner signing still completes gpg verification even if not all the desired deliverables are present and signed

Build Engineers

lsblakk - Tracking bug: bug 681451

Signed-off Revision(s)

Build 1: e5566280b79d

L10N changesets

Tags

Build # Branch, Tags Changeset
MOBILE70_2011092210_RELBRANCH, FENNEC_VERSION_RELEASE, FENNEC_VERSION_BUILD1 cb66cfac27f7
GECKO70_2011092208_RELBRANCH, FIREFOX_VERSION_BUILD1 FIREFOX_VERSION_RELEASE 01fc7213b1d8

Build data

Firefox

Build # Type Build ID Build machine Time to build
2 Linux 20110922153450 mv-moz2-linux-ix-slave09 2 hrs, 35 mins
Linux64 moz2-linux64-slave03 4 hrs, 32 secs
Mac moz2-darwin10-slave50 4 hrs, 18 mins
Windows mw32-ix-slave06 2 hrs, 44 mins

Fennec

Build # Type Build ID Build machine Time to build
2 android 20110922162525 mv-moz2-linux-ix-slave16 1 hr, 33 mins
maemo5-gtk 20110922152440 mv-moz2-linux-ix-slave06 48 mins
linux 20110922155923 mv-moz2-linux-ix-slave19 1 hr, 4 mins
macosx 20110922172123 bm-xserve09 2 hrs, 5 mins
win32 20110922163125 mw32-ix-slave02 1 hr, 10 mins

Notes

using buildbot-master07

Build 1

Synching the repos

  • synced mozilla-beta to mozilla-release (including l10n) according to documentation

There was an issue with the final push which require resolving a two head issue on a relbranch, this prevented pushing because of our singlehead hook resolved by doing: 

hg up -r GECKO60_2011080402_RELBRANCH
hg heads GECKO60_2011080402_RELBRANCH
hg merge
(adjusted for .hgtags conflict)
hg commit -m "Merging together weird heads on GECKO60_2011080402_RELBRANCH; doesn't affect any code that is still to be released, just needs to be done to get around the single head per branch hook"
hg up -r default
hg -R mozilla-release push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/mozilla-release

Preparing to start automation

  • landed configs - had to add a fix for relbranch = None (no longer can use empty string) - used l10n changesets from 7.0b7 (fennec, firefox) since there were a few updates since 7.0b6
  • landed mozharness bumps
  • tagged mozharness, tools, buildbotcustom, and buildbot-configs with FIREFOX_7_0_BUILD1 FIREFOX_7_0_RELEASE FENNEC_7_0_BUILD1 FENNEC_7_0_RELEASE
  • merged buildbot-configs default to production (buildbotcustom was already merged)
  • reserved 10 slaves
echo "10" > master/reserved_slaves_bm07-build1
  • clobbered "any master", "mozilla-release" and "any builder"
  • did a diff of mozilla-beta/release mozconfigs against mozilla-release/release mozconfigs for all release platforms - no diff, nothing committed
  • updated master
# in /builds/buildbot/build1
make update
  • release sanity:
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk \
 -V 7.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
 --dryrun --products firefox,fennec localhost:9001
  • failed:
2011-09-22 11:16:43,053 : ERROR : cannot find l10n dashboard at https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx7.0
2011-09-22 11:16:43,053  : ERROR : Error verifying l10n dashboard changesets
  • ignorable, since there is no dashboard entry for 7.0
  • edited release_sanity.py:
if not verify_l10n_dashboard(releaseConfig['l10nRevisionFile']):
    test_success = True
  • started automation:
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk \
 -V 7.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
 -b --products firefox,fennec localhost:9001
  • reverted change to release_sanity.py

E-mail Metrics

Only for beta1 and 1.9.2 releases

E-mail socorro-dev

Sent an email to socorro-dev@mozilla.com with the body "Build IDs will go here: https://wiki.mozilla.org/Releases/Firefox_7.0/BuildNotes#Build_data"

Tag

Fennec: no problems

Firefox: tagging failed on locales: mn,sw. There was an email to r-d from Axel on Aug 16th (more than a month prior to this release) mentioning having mn and sw in beta but not release however those locales were still copied over from beta (a note in the tracking bug might have helped with this) so they broke on tag attempt.

Checked in with Christian and Axel regarding whether mn,sw were supposed to ship (they are not) so removed them from shipped-locales (using linux-ix-slave01 where I had done the repo synching): 

cd mozilla-release/
hg pull && hg up -C
hg up -r GECKO70_2011092208_RELBRANCH
vim browser/locales/shipped-locales 
hg diff

[cltbld@linux-ix-slave01 mozilla-release]$ hg diff
diff -r fa92f7727fa1 browser/locales/shipped-locales
--- a/browser/locales/shipped-locales	Thu Sep 22 09:01:45 2011 -0700
+++ b/browser/locales/shipped-locales	Thu Sep 22 19:44:12 2011 -0700
@@ -52,7 +52,6 @@
 mai
 mk
 ml
-mn
 mr
 nb-NO
 nl
@@ -73,7 +72,6 @@
 sq
 sr
 sv-SE
-sw
 ta
 ta-LK
 te

hg commit -u "ffxbld <release@mozilla.com>" -m "Bug 681451 updating shipped-locales to remove mn, sw from 7.0 release"
hg tag -f -r GECKO70_2011092208_RELBRANCH FIREFOX_7_0_BUILD1 FIREFOX_7_0_RELEASE FENNEC_7_0_RELEASE FENNEC_7_0_BUILD1
hg out

[cltbld@linux-ix-slave01 mozilla-release]$ hg out
comparing with http://hg.mozilla.org/releases/mozilla-release
searching for changes
changeset:   73376:496a74e25ae8
branch:      GECKO70_2011092208_RELBRANCH
tag:         FENNEC_7_0_BUILD1
tag:         FENNEC_7_0_RELEASE
tag:         FIREFOX_7_0_BUILD1
tag:         FIREFOX_7_0_RELEASE
user:        ffxbld <release@mozilla.com>
date:        Thu Sep 22 19:46:36 2011 -0700
summary:     Bug 681451 updating shipped-locales to remove mn, sw from 7.0 release

changeset:   73377:273977a2c0ea
branch:      GECKO70_2011092208_RELBRANCH
tag:         tip
user:        cltbld
date:        Thu Sep 22 19:47:18 2011 -0700
summary:     Added tag FIREFOX_7_0_BUILD1, FIREFOX_7_0_RELEASE, FENNEC_7_0_RELEASE, FENNEC_7_0_BUILD1 for changeset 496a74e25ae8

cd ..
hg -R mozilla-release push -f -e "ssh -l ffxbld -i ~/.ssh/ffxbld_dsa" ssh://hg.mozilla.org/releases/mozilla-release

Note I had moved the Fennec tags off the MOBILE70_2011092210_RELBRANCH (thank you Rail!) and was able to move that tag back to the right place before restarting automation.

  • Added releaseConfig['skip_tag'] = True to the config on bm07, did make update on the master ran release_sanity again (with bypass on)
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk \
-V 7.0 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py \
-b --products firefox,fennec localhost:9001

This failed again (because make update does -C) so I re-added the skip_tag and did the above automation starting again. Second time worked fine.

Bouncer Submitter

No problems

Source

No problems

Build 2 called after tagging re-trigger created build with non-matching revisions (same code, but two relbranches)

Build 2

Synching repos

(eg: mozilla-beta -> mozilla-release for new Firefox releases) Already done for build2

Preparing to start Automation

  • clobberer set on any master, mozilla-release, any builder
  • updated configs and mozharness
  • tagged buildbotcustom, buildbot-configs, tools, mozharness with FIREFOX_7_0_BUILD2 and FENNEC_7_0_BUILD2 (as well as moving the _RELEASE tags where appropriate)
  • on buildbot-master07 (build1 dir)
make update
cd master/
source ../bin/activate
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk \
-V 7.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
--dryrun --products firefox,fennec localhost:9001
  • Still not passing on the l10n dashboard (we don't have milestones) so went ahead and started automation with -b to bypass that failure
PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk \
-V 7.0 --branch mozilla-release --build-number 2 -c release-firefox-mozilla-release.py \
-b --products firefox,fennec localhost:9001

E-mail Metrics

Only for beta1 and 1.9.2 releases

E-mail socorro-dev

Already done in build1

Tag

No problems

Bouncer Submitter

Already done for build1

Source

No problems

Start autosign

Copied ~/tmp/sign_6.0.sh to ~/tmp/sign_7.0.sh and updated it for 7.0 build2 

PRODUCT=firefox
VERSION=7.0
BUILD=2
TAG=FIREFOX_7_0_RELEASE
REPO='releases/mozilla-release'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-release.py

cd ${HOME}/hg-tools/
hg pull && hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

No problems

Fennec

Android

No problems

Maemo5-GTK

No problems

Desktop Builds

No problems

Desktop repacks

No problems

Unittests / Talos

XULRunner

Source

No problems

Build

No problems

Signing

# on keysign master
cd ~/tmp
cp sign_xr60.sh sign_xr70.sh
# edit the VERSION, BUILD, TAG
chmod +x sign_xr70.sh
./sign_xr70.sh
(enter pgp twice)

First signing said it succeeded but on inspection of stage, there were no win32 signed bits in the runtimes dir. Looks like it ran out of space during signing and somehow still passed verification even though it did not create signed win32 packages. This is definitely a flaw in the verification. Cleared up some room by removing xulrunner 7.0b{5,6} binaries and re-ran signing script. Filed bug 688764

Worked fine on second attempt once there was 4GB of space available but then the email showed that gpg fail on MD5SUMS and SHA1SUMS signatures: 

Verifying signed-build2/MD5SUMS... gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: armor header: Version: GnuPG v1.4.9 (Cygwin)
gpg: Signature made Fri Sep 23 07:53:38 2011 PDT using DSA key ID C52175E2
gpg: using subkey C52175E2 instead of primary key 1EBCAB3A
gpg: using PGP trust model
gpg: BAD signature from "Mozilla Software Releases <releases@mozilla.org>"
gpg: binary signature, digest algorithm SHA1

GPG verification FAILED
Verifying signed-build2/SHA1SUMS... gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: armor header: Version: GnuPG v1.4.9 (Cygwin)
gpg: Signature made Fri Sep 23 07:53:40 2011 PDT using DSA key ID C52175E2
gpg: using subkey C52175E2 instead of primary key 1EBCAB3A
gpg: using PGP trust model
gpg: BAD signature from "Mozilla Software Releases <releases@mozilla.org>"
gpg: binary signature, digest algorithm SHA1

MD5SUMS.asc and SHA1SUMS.asc from 7:53 am came in via unsigned dir on the second attempt, so I cleared off all .asc files on stage and did a clean re-signing again. Third time's a charm. All clear.

Repacks

Firefox

No problems

Fennec

No problems

Partner Repack

Firefox

No problems

Fennec

Forced release-mozilla-release-linux-maemo5-gtk_partner_repack builder, no problems

Firefox Signing

No problems

Android Signing

followed instructions - had to clobber and re-do signing due to bad c&p of passphrase. Second try was fine, automation sent out "All builds signed" before I had it up right so I filed bug 688669 on that.

Android Signature Verification

Red on en-US first time because of bad upload of signing, re-ran it manually with the url to the apk as 'who' property and it passed second time. Multi was fine.

Maemo5-GTK repos

No problems.

L10N verify

No problems

Updates

Failed uploading missing aus2.beta directory. TODO: file a bug which disables this step if we don't generate beta snippets.

Sent an email to av-vendor-release-announce.

Update verify

# On my local machine, to kick off update verify
for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    curl "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F10/force" >/dev/null
    sleep 5
  done
done

Major Update

Update verify

Socorro Product Data Updates

N/A since there is no version change

Check permissions / AV scan

Forced builds on check permissions and AV builders, no problems.

Reset reserved slaves

echo 0> master/reserved_slaves_bm07-build1

Push files to internal mirrors

updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to add 

- firefox/releases/7.0

wait 10 minutes (8:16 PDT) 

# in /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-7.0-build2 2>&1 | tee lsblakk.out

real    66m22.139s
user    0m8.160s
sys     0m38.939s

force the release_push_to_mirrors with the following: 

release_config: mozilla/release-firefox-mozilla-release.py
script_repo_revision: FIREFOX_7_0_RELEASE

Push to beta

Nothing to do here for this release

Push XULRunner to Mirrors

ssh -l xrbld -i ~/.ssh/xrbld_dsa stage.mozilla.org

# DRY RUN
rsync -n -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/7.0-candidates/build2/ /pub/mozilla.org/xulrunner/releases/7.0/
# FOR REAL
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/7.0-candidates/build2/ /pub/mozilla.org/xulrunner/releases/7.0/

Update XULRunner wiki page

Updated Gecko_SDK wiki page

Push index file to mirrors

Added the following lines to the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude: 

+ firefox
+ firefox/releases
+ firefox/releases/7.0
+ firefox/releases/7.0/*/
+ firefox/releases/7.0/*/*/
+ firefox/releases/7.0/*/*/*/
+ firefox/release/7.0/**index.html
- firefox/releases/7.0/**

Waited 10 minutes to ensure the rsync server picks up the changes, then removed the previously added "- firefox/releases/7.0" line. After that, ran the following to lay out the index files: 

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases/7.0
wget --no-check-certificate -O index.html https://bugzilla.mozilla.org/attachment.cgi?id=540139
sed -i -e 's/5/7/g' index.html 
for dir in `find . -type d -mindepth 1`; do cp -pv index.html $dir/; done

emailed mirrors@mozilla.org regarding the releases (this has been added to the notes template so as to happen sooner).

Final verification

Linux failed on a couple of links, I checked them manually and they were resolving so I forced the builder and the second run went green. The rest were green on the first time through.

Push to external mirrors

Removed the following lines to the top of stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude: 

+ firefox
+ firefox/releases
+ firefox/releases/7.0
+ firefox/releases/7.0/*/
+ firefox/releases/7.0/*/*/
+ firefox/releases/7.0/*/*/*/
+ firefox/releases/7.0/**index.html
- firefox/releases/7.0/**

Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to carry 7.0 instead of 6.0.2

Went to bed as mirrors were picking up, in the morning forced build on the release_ready_for_release builder to send the email that mirror uptake was sufficient for release.

Publish Fennec

Create bouncer entry for maemo

Added a Firefox-Mobile-7.0 product in bouncer and added /mobile/releases/maemo/:lang/firefox.install to the maemo location

Push the files

Ran the following script on stage as ffxbld (~/push_fennec_release.sh): 

export VERSION=7.0
export BUILDNUM=2
export RD=/home/ftp/pub/mozilla.org/mobile/releases
export CD=/home/ftp/pub/mozilla.org/mobile/candidates

mkdir $RD/$VERSION
cd $RD/$VERSION

rsync -av --exclude=unsigned --exclude=*.txt --exclude=*crashreporter* \
 --exclude=*tests* --exclude=*unaligned* --exclude=*old* --exclude=*install* \
 --exclude=*logs* \
 $CD/$VERSION-candidates/build$BUILDNUM/ $RD/$VERSION/

for p in linux macosx win32; do
    locales=`cd $CD/$VERSION-candidates/build$BUILDNUM/$p/; ls -1 --hide=en-US`
    for i in $locales; do
        cp -a $CD/$VERSION-candidates/build$BUILDNUM/$p/$i/install/*.xpi \
          $RD/$VERSION/$p/$i
    done
done

# repos
mkdir $RD/$VERSION/repos
rsync -av --exclude=*.install $CD/$VERSION-candidates/repos/ $RD/$VERSION/repos/

# symlink repos
rm -f maemo
ln -s repos maemo

# link fremantle to chinook for multi deb repo
cd $RD/$VERSION/maemo/multi/dists
ln -s fremantle chinook

cd $RD
rm -f latest
ln -s $VERSION latest

Publish to the Android Market

IMPORTANT - Make sure to follow instructions! See 6.0b6's push to Market.

Push to Release Channel

 # cltbld@aus2-staging
 time ~/bin/pushsnip Firefox-7.0-build2

sent 1786217 bytes  received 101970 bytes  10289.85 bytes/sec
total size is 1418919  speedup is 0.75

real    3m3.605s
user    0m0.792s
sys     0m3.074s

Running ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com touch /opt/aus2/incoming/3
---------------------------------------------------------------------------

real    15m9.481s
user    0m2.496s
sys     0m13.512s

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
ln -s 7.0 latest-7.0
rm latest && ln -s latest-7.0 latest

Remove index.html files

cd /pub/mozilla.org/firefox/releases/7.0
find . -name index.html -exec echo rm -v {} \;
find . -name index.html -exec rm -v {} \;

Update browser choice website

Filed bug 689576 to update the website


Turning off updates for bug 680802

# cltbld@aus2-staging
cd /opt/aus2/incoming/3/Firefox
chmod 0 4.0* 5.0* 6.0* 7.0*
touch .

# and phx too
ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com
cd /opt/aus2/incoming/3/Firefox
chmod 0 4.0* 5.0* 6.0* 7.0*
touch .

The rest of this is tracked in bug 689906.

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_7.0/BuildNotes&oldid=352752"