Releases/Firefox 7.0.1/BuildNotes

From MozillaWiki
Jump to: navigation, search

Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Enter any bugs pre-existing or newly discovered and filed during the release:

  • bug 670951 - Mac XULRunner builds fail intermittently during unification (reopened but this is fixed and just not going to be present until 9.0)
  • bug 689906 - re-enable 7.0 updates when 7.0.1 is released
  • bug 690698 - Mac/Linux partner repacks are missing
  • bug 690730 - MD5SUMS/SHA1SUMS missing linux/mac
  • bug 690809 - check against unsigned too when verifying asc files
  • bug 691369 - makefile needs a check-downloads target
  • bug 690785 - release sanity should verify that shipped-locales doesn't have any locales that aren't in l10n-changesets

Build Engineers

lsblakk - Tracking bug: bug 690016

Signed-off Revision(s)

Build 1: a494b8cd79bf

L10N changesets

(re-using 7.0, minus mn and sw in Firefox)

Tags

Build # Branch, Tags Changeset
1 MOBILE701_2011092812_RELBRANCH, FENNEC_VERSION_RELEASE, FENNEC_VERSION_BUILD1 096fff58d19c
GECKO70_2011092812_RELBRANCH, FIREFOX_VERSION_BUILD1 FIREFOX_VERSION_RELEASE 907a687be34f

Build data

Firefox

Build # Type Build ID Build machine Time to build
1 Linux 20110928134238 mv-moz2-linux-ix-slave09 2 hrs, 33 mins
Linux64 moz2-linux64-slave01 2 hrs, 47 mins
Mac moz2-darwin10-slave29 4 hrs, 20 mins
Windows mw32-ix-slave13 2 hrs, 44 mins

Fennec

Build # Type Build ID Build machine Time to build
1 android 20110928142707 mv-moz2-linux-ix-slave16 1 hr, 37 mins
maemo5-gtk 20110928140603 mv-moz2-linux-ix-slave06 1 hr, 32 mins
linux 20110922155923 mv-moz2-linux-ix-slave19 1 hr, 4 mins
macosx 20110928152044 bm-xserve09 2 hrs, 6 mins
win32 20110928154525 mw32-ix-slave15 1 hr, 14 mins

Notes

Build 1

Syncing repos

(eg: mozilla-beta -> mozilla-release for new Firefox releases) N/A

Preparing to start Automation

  • reserved slaves
echo "10" > master/reserved_slaves_bm07-build1 
  • clobbered any master, mozilla-release, any builder
  • no new l10n - chemspill - the current ones have mn,sw removed for Firefox Desktop
  • landed configs, mozharness bumps
  • merged default >> production for buildbot{custom,-configs}
  • tagged tools, mozharness, buildbotcustom, buildbot-configs with FIREFOX_7_0_1_BUILD1, FIREFOX_7_0_1_RELEASE, FENNEC_7_0_1_BUILD1, FENNEC_7_0_1_RELEASE
 make update (on buildbot-master07 /builds/buildbot/build1)
 cd master/
 . ../bin/activate

dryrun release_sanity

 PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 7.0.1 \
 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py -c \
 release-fennec-mozilla-release.py --dryrun --products firefox,fennec localhost:9001 

only error about missing l10n dashboard, expected as there isn't one, started automation

 # using -b to bypass
 PYTHONPATH=. ../bin/python ../tools/buildbot-helpers/release_sanity.py -u lsblakk -V 7.0.1 \
 --branch mozilla-release --build-number 1 -c release-firefox-mozilla-release.py -c \
 release-fennec-mozilla-release.py -b --products firefox,fennec localhost:9001 

E-mail Metrics

Only for beta1 and 1.9.2 releases

E-mail mirrors@mozilla.org

Emailed mirrors to let them know that we're on a chemspill that will be pushed in the next 24 hrs.

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

  • cleaned up ~30 GB of space in D:
  • copied ~/tmp/sign_7.0.sh to ~/tmp/sign_7.0.1.sh
bash -xe sign_7.0.1.sh 2>&1 | tee sign_7.0.1.log

Build

Firefox

No problems

Fennec

Android

No problems

Maemo5-GTK

No problems

Desktop Builds

No problems

Desktop repacks

No problems

Unittests / Talos

XULRunner

Source

No problems

Build

Ran into bug 670951, reopened it, then forced rebuild on the macosx64 xulrunner. Second build also failed compile. Third time was a charm.

Signing

cp ~/tmp/sign_xr7.0.sh ~/tmp/sign_xr_7.0.1.sh
bash -xe sign_7.0.1.sh 2>&1 | tee sign_7.0.1.log 

No problems

Repacks

Firefox

All platforms: repacks 4/6 & 6/6 failed because mn & sw made it back into [hg.mozilla.org/releases/mozilla-release/file/FIREFOX_7_0_1_RELEASE/browser/locales/shipped-locales shipped-locales]

Fennec

No problems

Partner Repack

Firefox

Manually triggered all because of repack fail (missing mn,sw locales), no problems

note - later on when I updated the shipped-locales in mozilla-release, the partner repacks were automatically triggered, now that all locales were present, so I had to kill them. no apparent issues caused by this.

Fennec

Forced release-mozilla-release-linux-maemo5-gtk_partner_repack builder, no problems

Firefox Signing

Because we were missing mn and sw (according to shipped-locales) I stopped the download loop once it had gotten all the partner repacks downloaded. Then I opened up a new cywin window and did:

make -f Makefile PRODUCT=firefox VERSION=7.0.1 BUILD=1 TAG=FIREFOX_7_0_1_RELEASE \
REPO='releases/mozilla-release' EMAIL=release@mozilla.com KEYDIR=d:/2010-keys \
RELCONFIG=mozilla/release-firefox-mozilla-release.py sign stage verify fake-upload postsign

See the bugs hit section - there were missing non-windows sigs from the final deliverables so I went back on to keymaster and did the following:

# in signed-build1 copied over non-win32
cp -r ../unsigned-build1/linux-* .
cp -r ../unsigned-build1/mac* .
cp -r ../unsigned-build1/source .

# copy over updates
cp -r ../unsigned-build1/update/{linux-i686,linux-x86_64} update/

# copy partner-repacks
for dir in `ls ../unsigned-build1/unsigned/partner-repacks`; do cp -r ../
unsigned-build1/unsigned/partner-repacks/$dir/{linux-i686,mac,linux-x86_64} part
ner-repacks/$dir/; done

Run make again to do just checksum-files

make -f Makefile PRODUCT=firefox VERSION=7.0.1 BUILD=1 TAG=FIREFOX_7_0_1_RELEASE \
REPO='releases/mozilla-release' EMAIL=lsblakk@mozilla.com KEYDIR=d:/2010-keys \
RELCONFIG=mozilla/release-firefox-mozilla-release.py checksum-files

I checked that the new MD5SUMS still matched the win32 ones, then did the above make call but to fix-permissions, then verify-asc, then discovered that I needed to do create-sigs before I could verify them. Compared the word count of MD5SUM from 7.0build2 to the newly generated one

cat MD5SUMS | wc -l
# matched with 1031 lines

Ran create-sigs

make -f Makefile PRODUCT=firefox VERSION=7.0.1 BUILD=1 TAG=FIREFOX_7_0_1_RELEASE \
REPO='releases/mozilla-release' EMAIL=lsblakk@mozilla.com KEYDIR=d:/2010-keys \
RELCONFIG=mozilla/release-firefox-mozilla-release.py create-sigs

Clean up & try again

The timestamps were different when I tested fake-upload (because of cp instead of rsync) so I cleaned up and started again:

# in signed-build1
rm -rf {linux*,mac,source}
rm *.asc
for dir in `ls partner-repacks`; do echo rm -rf partner-repacks/$dir/{linux-i686,mac,linux-x86_64}; done
for dir in `ls partner-repacks`; do rm -rf partner-repacks/$dir/{linux-i686,mac,linux-x86_64}; done
rm -rf update/{linux*,mac}

Now do it with rsync

# builds & source
rsync -n -av ../unsigned-build1/{linux-*,mac,source} .
rsync -av ../unsigned-build1/{linux-*,mac,source} .
# updates
rsync -n -av ../unsigned-build1/update/{linux-i686,linux-x86_64} update/
rsync -av ../unsigned-build1/update/{linux-i686,linux-x86_64} update/
# partner repacks
for dir in `ls ../unsigned-build1/unsigned/partner-repacks`; do rsync -n -av \
../unsigned-build1/unsigned/partner-repacks/$dir/{linux-i686,mac,linux-x86_64} \
partner-repacks/$dir/; done
for dir in `ls ../unsigned-build1/unsigned/partner-repacks`; do rsync -av \
../unsigned-build1/unsigned/partner-repacks/$dir/{linux-i686,mac,linux-x86_64} \
partner-repacks/$dir/; done

Now do make again

cd ..
make -f Makefile PRODUCT=firefox VERSION=7.0.1 BUILD=1 TAG=FIREFOX_7_0_1_RELEASE \
REPO='releases/mozilla-release' EMAIL=lsblakk@mozilla.com KEYDIR=d:/2010-keys \
RELCONFIG=mozilla/release-firefox-mozilla-release.py create-sigs checksum-files verify-asc fake-upload
make -f Makefile PRODUCT=firefox VERSION=7.0.1 BUILD=1 TAG=FIREFOX_7_0_1_RELEASE \
REPO='releases/mozilla-release' EMAIL=lsblakk@mozilla.com KEYDIR=d:/2010-keys \
RELCONFIG=mozilla/release-firefox-mozilla-release.py upload

custom fake-upload to test

rsync -n -i -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -avO \
/home/cltsign/signing-work/firefox-7.0.1/signed-build1/ \
ffxbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/7.0.1-candidates/build1/ \
| grep -v exe | grep -v 'update/win32/.*/.*\.mar' | pastebin

And to push the files to releases:

# ffxbld@stage
rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=logs --exclude=jsshell* /pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/ /pub/mozilla.org/firefox/releases/7.0.1/

Android Signing

Followed instructions

Android Signature Verification

No problems

Maemo5-GTK repos

Forced deb_repos builder, no problems

L10N verify

Updates

Updates failed because mn & sw were in the patcher configs so I went to the slave that had the configs checked out, removed the two locales and then committed

[cltbld@mv-moz2-linux-ix-slave17 patcher-configs]$ cvs commit -m 'Removing mn,sw from 7.0.1 patcher configs'
cvs commit: Examining .
Checking in mozRelease-branch-patcher2.cfg;
/cvsroot/mozilla/tools/patcher-configs/mozRelease-branch-patcher2.cfg,v  <--  mozRelease-branch-patcher2.cfg
new revision: 1.12; previous revision: 1.11
done

Next run still failed because configs had already been bumped, I commented out these parts of the updates factory, reconfig'd buildbot-master07 and rebuilt

#self.bumpPatcherConfig()
#self.bumpVerifyConfigs()

Next run failed because it needed a clobber, clobbered and restarted. This time it was still going to look for the same amount of mars and it turned out that the earlier rebuild has also re-committed the locales back into patcher configs. Did the above steps to remove them again, committed to cvs, then clobbered and forced a build.

Current run of updates builder looks good, 664 downloads instead of 672 so its got the right config. Updates take more than 2 hours so I'm off to bed and will check in on them in the morning.

That run of updates failed when it wrongly tried to upload aus2.beta snippets, which (rightly) don't exist. Additionally, the second bump of the patcher config that the automation did caused filenames to be wrong in the betatest snippets, and in the actual partial MARs them:

Index: mozRelease-branch-patcher2.cfg
===================================================================
RCS file: /cvsroot/mozilla/tools/patcher-configs/mozRelease-branch-patcher2.cfg,v
retrieving revision 1.11
retrieving revision 1.13
diff -u -r1.11 -r1.13
--- mozRelease-branch-patcher2.cfg	29 Sep 2011 08:08:48 -0000	1.11
+++ mozRelease-branch-patcher2.cfg	29 Sep 2011 09:03:45 -0000	1.13
@@ -4,17 +4,17 @@
             beta-dir   beta
             channel   release
             <complete>
-                beta-url   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.complete.mar
-                betatest-url   http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.complete.mar
-                path   firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.complete.mar
+                beta-url   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.1.complete.mar
+                betatest-url   http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.1.complete.mar
+                path   firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0.1.1.complete.mar
                 url   http://download.mozilla.org/?product=firefox-7.0.1-complete&os=%bouncer-platform%&lang=%locale%
             </complete>
             details   https://www.mozilla.com/%locale%/firefox/7.0.1/releasenotes/
             from   7.0
             <partial>
-                beta-url   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.partial.mar
-                betatest-url   http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.partial.mar
-                path   firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.partial.mar
+                beta-url   http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.1.partial.mar
+                betatest-url   http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.1.partial.mar
+                path   firefox/nightly/7.0.1-candidates/build1/update/%platform%/%locale%/firefox-7.0-7.0.1.1.partial.mar
                 url   http://download.mozilla.org/?product=firefox-7.0.1-partial-7.0&os=%bouncer-platform%&lang=%locale%
             </partial>
             <rc>

There's nothing we need to do about the former issue except force subsequent automation along. For the latter, had to fix the snippets & patcher config manually.

To fix the filenames:

# ffxbld@stage
cd /home/ftp/pub/firefox/nightly/7.0.1-candidates/build1/update
find . -name "*.partial.mar" -exec rename 7.0.1.1 7.0.1 {} \;

For the snippets:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
rsync -av Firefox-7.0.1-build1-test/ Firefox-7.0.1-build1-test.bad/
# releasetest is unaffected
find Firefox-7.0.1-build1-test -type f -wholename "*betatest*" -exec sed -i -e 's,7\.0\.1\.1,7.0.1,' {} \;
~/bin/pushsnip Firefox-7.0.1-build1-test


And the patcher config will be fixed in bug 690016.

While pushsnip was running, forced the permissions check and antivirus builders. Didn't force update verify (need to wait until 6.0.2 -> 7.0.1 partials are ready)

Also backed out the local change to buildbotcustom (but didn't do a reconfig)

6.0.2 -> 7.0.1 partial MARs

First, created backups of the existing snippets, because we'll be merging right into the existing directories:

# cltbld@aus2-staging
rsync -av Firefox-7.0.1-build1-test{,.bak}/
rsync -av Firefox-7.0.1-build1{,.bak}/
cd /builds/slave/rel-m-rel-updates/build

wget -O patcher-configs/mozRelease-branch-patcher2.cfg https://bugzilla.mozilla.org/attachment.cgi?id=563389

perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-branch-patcher2.cfg
perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/mozRelease-branch-patcher2.cfg

find temp/firefox/6.0.2-7.0.1/ftp/firefox/nightly/7.0.1-candidates/build1 -type f -exec chmod 644 '{}' ';'
find temp/firefox/6.0.2-7.0.1/ftp/firefox/nightly/7.0.1-candidates/build1 -type d -exec chmod 755 '{}' ';'

rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' temp/firefox/6.0.2-7.0.1/ftp/firefox/nightly/7.0.1-candidates/build1/update ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/7.0.1-candidates/build1/

rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' --exclude=complete.txt temp/firefox/6.0.2-7.0.1/aus2.test/Firefox/6.0.2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-7.0.1-build1-test/Firefox/6.0.2/
rsync -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' --exclude=complete.txt temp/firefox/6.0.2-7.0.1/aus2/Firefox/6.0.2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-7.0.1-build1/Firefox/6.0.2/

And re-pushed the test snippets:

# cltbld@aus2-staging
~/bin/pushsnip Firefox-7.0.1-build1-test

Also, manually added a bouncer entry for Firefox-7.0.1-Partial-6.0.2

Update verify

upload_snippets2 failed because of missing aus2.beta, filed bug 690365

As per bug 689906 we had to turn updates back on before being able to verify them (my first trigger of the builders all failed right away). But we have to keep the release snippets offline for now.

On the aus2-staging and in phx aus ran:

# on aus2-staging
cd /opt/aus2/incoming/3/Firefox
for dir in 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0; do
  find $dir -type d -name release -exec echo {} \;
  find $dir -type d -name release -exec chmod 700 {} \;
done 2>&1 | tee turnoff_release.log
chmod 775 4.0* 5.0* 6.0* 7.0*
touch .

# and phx too
ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com
cd /opt/aus2/incoming/3/Firefox
for dir in 4.0 4.0.1 5.0 5.0.1 6.0 6.0.1 6.0.2 7.0; do
  find $dir -type d -name release -exec echo {} \;
  find $dir -type d -name release -exec chmod 700 {} \;
done 2>&1 | tee turnoff_release.log
chmod 775 4.0* 5.0* 6.0* 7.0*
touch .

Now kick off update verify

# On my local machine, to kick off update verify
for p in linux linux64 macosx64 win32; do
  for i in 1 2 3 4 5 6 7 8 9 10; do
    curl "http://buildbot-master07.build.sjc1.mozilla.com:8001/builders/release-mozilla-release-${p}_update_verify_${i}%2F10/force" >/dev/null
    sleep 5
  done
done

Check permissions / AV scan

No problems

Push files to internal mirrors

Updated rsyncd-mozilla-releases.exclude with the following line:

- firefox/releases/7.0.1

Waited 10 minutes for the file to sync around.

# in /opt/aus2/snippets/staging
~/bin/backupsnip Firefox-7.0.1-build1 2>&1 | tee lsblakk.out
real    57m13.473s
user    0m9.760s
sys     0m47.489s

While waiting for backupsnip, I updated shipped-locales so that this issue doesn't bite the next release:

on relbranch to make sure update verify works for the next release
on default to make sure we don't accidentally build mn and sw if we have a 7.0.2 
# on a slave
hg clone http://hg.mozilla.org/releases/mozilla-release mozilla-release
cd mozilla-release/
hg up -C -r GECKO701_2011092812_RELBRANCH
vim browser/locales/shipped-locales 
hg commit -m "Removing mn,sw from shipped-locales as we don't ship these beyond beta channel for now"
hg log -l2
hg tag -f -r a6c276265898 FIREFOX_7_0_1_RELEASE
hg up -r default
vim browser/locales/shipped-locales 
hg commit -m "Removing mn,sw from shipped-locales as we don't ship these beyond beta channel for now"
hg out -p
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' ssh://hg.mozilla.org/releases/mozilla-releas

Reset reserved slaves

echo "0" > master/reserved_slaves_bm07-build1

Final verification

No problems

Push XULRunner to Mirrors

# DRYRUN
rsync -n -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/7.0.1-candidates/build1/ /pub/mozilla.org/xulrunner/releases/7.0.1/
# FOR REAL
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip --exclude=*.checksums \
/pub/mozilla.org/xulrunner/nightly/7.0.1-candidates/build1/ /pub/mozilla.org/xulrunner/releases/7.0.1/

Update XULRunner wiki page

Updated the 7.0 download on wiki page

Push to external mirrors

  • Removed the firefox/releases/7.0.1 line from stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude
  • Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to 7.0.1

Publish Fennec

Create bouncer entry for maemo

Added /mobile/releases/maemo/:lang/firefox.install for Firefox-Mobile-7.0.1

Push the files

Updated the ~ffxbld/push_fennec_release.sh script on stage & ran without issue

Publish to the Android Market

IMPORTANT - Make sure to follow instructions! See 6.0b6's push to Market.

Push to Release Channel

# cltbld@aus2-staging
time ~/bin/pushsnip Firefox-7.0.1-build1
sent 2037569 bytes  received 133480 bytes  25392.39 bytes/sec
total size is 1608545  speedup is 0.74

real    1m24.886s
user    0m0.715s
sys     0m2.120s

Running ssh -i /home/cltbld/.ssh/auspush ffxbld@dp-ausstage01.phx.mozilla.com touch /opt/aus2/incoming/3
---------------------------------------------------------------------------

real    9m44.349s
user    0m2.655s
sys     0m11.217s

Update symlinks

Once we're signed off on the release channel.

# as ffxbld@stage in /pub/mozilla.org/firefox/releases
rm latest-7.0 && ln -s 7.0.1 latest-7.0

Update browser choice website

Filed bug 690578 for updating the site.