Releases/Firefox 3.0.16/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineer(s)

coop

Tracking release bug

Bonsai queries

Tags

Build 1:

Module Branch Tag Pull date
cvsroot/mozilla HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD1 FIREFOX_3_0_16_RELEASE 2009-11-19 10:28 PDT (wrong timezone!)
l10n/l10n HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD1 FIREFOX_3_0_16_RELEASE 2009-09-22 05:34 PDT

Build 2:

Module Branch Tag Pull date
cvsroot/mozilla HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD2 FIREFOX_3_0_16_RELEASE 2009-11-30 18:29 PST
l10n/l10n HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD2 FIREFOX_3_0_16_RELEASE 2009-09-22 05:34 PDT

Build 3:

Module Branch Tag Pull date
cvsroot/mozilla HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD3 FIREFOX_3_0_16_RELEASE 2009-12-01 15:18 PST
l10n/l10n HEAD GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD3 FIREFOX_3_0_16_RELEASE 2009-09-22 05:34 PDT

Notes

Build 1

  • on fx-linux-1.9-slave2 deleted:
    • /builds/verify/firefox-3.0.15
    • /builds/verify/firefox-3015-355-major
    • /builds/source/firefox-3.0.15
    • /builds/data/cltbld/firefox-3.0.15
    • /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
    • /builds/tinderbox/Fx-Mozilla1.9-l10n-Release//Linux_2.6.18-53.1.13.el5_Depend
  • On fx-mac-1.9-slave2 deleted:
    • /builds/verify/firefox-3.0.15
    • /builds/verify/firefox-3015-354-major
    • /builds/verify/firefox-3015-355-major
  • On fx-win32-1.9-slave2 deleted:
    • /e/xr19rel/WINNT_5.2_Depend
    • /e/fx19l10nrel/WINNT_5.2_Depend
    • /e/fx19rel/WINNT_5.2_Depend
  • Available space on slaves:
    • fx-linux-1.9-slave2 35 GB on /builds
    • fx-mac-1.9-slave2 17 GB on /
    • fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
  • Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build1 release" DOOOOOEEEEET!

Tag

Two aborted tag attempts prior to a successful one. Forgot to reconfig the master the first time, and then forgot to update the RELEASE_AUTOMATION_M15 tag on the bootstrap configs to point to the latest version.

Source

No problems.

Build & Repack

  • Linux - No problems
  • Mac - No problems
  • Win32 - *** RESPIN CALLED FOR bug 529401 ***

Build 2

  • on fx-linux-1.9-slave2 deleted:
    • nothing
  • On fx-mac-1.9-slave2 deleted:
    • /builds/partners/partner-repacks/scripts/original_builds
    • /builds/partners/partner-repacks/scripts/repacked_builds
  • On fx-win32-1.9-slave2 deleted:
    • /e/fx19l10nrel/WINNT_5.2_Depend
    • /e/fx19rel/WINNT_5.2_Depend
  • Available space on slaves:
    • fx-linux-1.9-slave2 31 GB on /builds
    • fx-mac-1.9-slave2 17 GB on /
    • fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
  • Update the bootstrap configs
  • Check them in and tag them or tag will fail
  • Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build2 release" DOOOOOEEEEET!

Tag

  • Forgot to tag the bumped configs, so the first tag run failed.
  • Discovered that the build 1 timestamp specified PDT instead of PST, so we missed the last change when creating the relbranch
  • interrupted source and en-US builds
  • merged the change over to the branch, creating rev 3.216.4.1, and moved the FIREFOX_3_0_16_RELEASE and FIREFOX_3_0_16_BUILD2 tags onto that
  • patched production-1.9-master:/build/buildbot/master.cfg and reconfiged
Index: master.cfg
===================================================================
RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v
retrieving revision 1.56
diff -u -r1.56 master.cfg
--- master.cfg	5 Aug 2009 03:43:58 -0000	1.56
+++ master.cfg	1 Dec 2009 03:11:29 -0000
@@ -449,13 +449,17 @@
  haltOnFailure=1,
 )
 
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.dummy import Dummy
+dummyFactory = BuildFactory()
+dummyFactory.addStep(Dummy())
 c['builders'].append(
  { 
   'name': 'tag',
   'slavename': 'fx-linux-1.9-slave2',
   'category': 'release',
   'builddir': 'tag',
-  'factory': tagFactory,
+  'factory': dummyFactory,
  },
 )
  • cleaned up fx-linux-1.9-slave2:/builds/source/firefox-3.0.16/batch-source/build2
  • cleaned up /builds/config/firefox-3.0.16-build2 on all three slaves
  • removed FIREFOX_3_0_16_BUILD2_l10n and FIREFOX_3_0_16_BUILD2 tags from mozilla/tools/tinderbox-configs/firefox
  • stopped buildbot on win32 slave, clean up link/make/perl processes, start buildbot
  • give another sendchange
buildbot sendchange --username=nthomas --master=localhost:9989 -m"Firefox 3.0.16build2 release - restart" DOOOOOEEEEET!

In retrospect, merging the change to the relbrnach and doing build 3 would have been much easier!

Source

No problems

Build & Repack

  • Linux - No problems
  • Mac - No problems
  • Win32 - No problems

Sign

L10nVerify

  • No problems

Generate Updates

  • No problems

Publish Updates to Test Channels (betatest & releasetest)

Update Verify

  • linux - No problems
  • win32 - .chk files problems (EXPECTED)
  • mac - No problems

Stage

No problems

*** RESPIN CALLED FOR bug 504862 ***

Build 3

  • on fx-linux-1.9-slave2 deleted:
    • /builds/source/firefox-3.0.16
    • /builds/updates/firefox-3.0.16
    • /builds/verify/firefox-3.0.16
    • /builds/tags/FIREFOX_3_0_16_BUILD1
    • /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
    • /builds/tinderbox/Fx-Mozilla1.9-l10n-Release/Linux_2.6.18-53.1.13.el5_Depend
  • On fx-mac-1.9-slave2 deleted:
    • nothing
  • On fx-win32-1.9-slave2 deleted:
    • /e/fx19l10nrel/WINNT_5.2_Depend
    • /e/fx19rel/WINNT_5.2_Depend
    • /e/xr19rel/WINNT_5.2_Depend
  • Available space on slaves:
    • fx-linux-1.9-slave2 29 GB on /builds
    • fx-mac-1.9-slave2 24 GB on /
    • fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
  • reverted dummy factory change in master.cfg on production-1.9-master
  • Updated the bootstrap configs
  • Checked them in and tagged them with RELEASE_AUTOMATION_M15, otherwise tagging will fail.
  • Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build3 release" DOOOOOEEEEET!
  • forgot to reconfig master, shutdown builds as quickly as possible, reconfig-ed , and then sent another sendchange

Tag

  • No problems

Source

  • No problems

Build & Repack

  • 1st attempt - The aborted first run got as far as Tinderconfig before I could kill it. :/
    • Linux - failed in Tinderconfig
    • Mac - failed in Tinderconfig
    • Win32 - slave lost???
  • 2nd attempt, with Tinderconfig step commented out:
    • Linux - No problems
    • Mac - No problems
    • Win32 - slow, but no problems

Sign

L10nVerify

  • No problems

Generate Updates

  • No problems

Publish Updates to Test Channels (betatest & releasetest)

  • No problems

Update Verify

  • linux - No problems
  • win32 - .chk files problems (EXPECTED)
  • mac - No problems

Stage

  • solaris builds had been uploaded to contrib/ already, so rsync "failed" to sync contrib/ due to permissions issues, but everything else synced fine.

XULRunner

  • kicked off by hand due to stage step "failing." No problems

Push updates to beta channel

# cltbld@aus2-staging.m.o
# make sure scripts are up to date
cd bin
cvs update
cd /opt/aus2/snippets/staging/
~/bin/backupsnip 20091202-Firefox-3.0.16-beta

Then on official "go" email do pushsnip:

~/bin/pushsnip 20091202-Firefox-3.0.16-beta

3.0.16 -> 3.5.6 Major Update Refresh

Snippet Generation

On fx-linux-1.9-slave2:

mkdir -p /builds/3.0.16-3.5.6-major-update/snippets
cd /builds/3.0.16-3.5.6-major-update/snippets

cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.5.6 3.0.16-3.5.6
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verify

Check that releasetest = beta = release. New in 3.0.15 we now use a new dir for beta

cd temp/firefox/3.0.16-3.5.6
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'
# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'
Push to AUS

New in 3.0.15 - 3 dirs to sync now instead of 2

cd /builds/3.0.16-3.5.6-major-update/snippets/patcher/temp/firefox/3.0.16-3.5.6
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU-beta
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU
Enable test snippets
ssh -i ~/.ssh/aus2 cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20091202-Firefox-3.0.16-3.5.6-MU-test
~/bin/pushsnip   20091202-Firefox-3.0.16-3.5.6-MU-test
Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3016-356-major/
cd /builds/verify/firefox-3016-356-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544§
patch -p3 < patch

# everyone ...
cd ../updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Check logs for results - whatever is after "succeeded", known issues are here

Verify succeeded if in the results everything is the same as 3.0.16 -> 3.5.6 MU. aka, everything is OK

Sign Installers

  • cleared some space on keymaster:
$ rm -rf xulrunner-1.9.0.13 xulrunner-1.9.0.14 xulrunner-1.9.1.[1-3] byob-3.5.3
  • firefox-3.5.3* firefox-3.6a*
  • Done manually on the day before release using these installer-signing-instructions here

On Stage complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.16/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.16/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

Done.

Wait for official "push to mirrors" email...

Push to mirrors

Make sure that you use ffxbld for 'push to mirrors' , rather than cltbld (this is a change, but a desired one).

  • push the stage-merged directory to the releases area. This can take a while, and can take even longer if there are contrib builds (solaris) in the subdirs already. For this reason, I sync the win32 builds first, and then perform a general rsync to catch everything after. This allows mirrors to start picking up the win32 builds (our most used platform) a little quicker:
# on stage
mkdir -p /home/ftp/pub/firefox/releases/3.0.16/win32
rsync -av /data/cltbld/firefox-3.0.16/stage-merged/win32/ /home/ftp/pub/firefox/releases/3.0.16/win32/
rsync -av /data/cltbld/firefox-3.0.16/stage-merged/ /home/ftp/pub/firefox/releases/3.0.16/
  • edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.16) and remove the previous release (3.0.15).

Final Verification

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
    • No Problems
  • Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20091201-Firefox-3.0.16
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20091201-Firefox-3.0.16-test/$a");'
  • this should have been 20091201-Firefox-3.0.16

Publish Updates to Release Channel

Start doing backupsnip a few hours at least before the release time

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20091201-Firefox-3.0.16
  • this should have been 20091202-Firefox-3.0.16
  • No problems

*** PROBLEM FOUND WITH UPDATES ON RELEASETEST ***

The solaris rsync problems in the Stage step were a bigger problem than I thought. The step exited early, and consequently build2 packages and mars were sent to mirrors instead of build3.

We deleted the releases/3.0.16 directory on stage.mozilla.org when we discovered this to prevent any further syncing. We realize that some users who checked manually may have found these installers. We plan to offer an update path from build2 sometime later this week.

In the meantime, we are going to start over from the Stage step (excluding the solaris builds), generate new snippets, re-sign installers, etc. We'll create a new product in bouncer, e.g. 3.0.16-real, and then do all the switching heroics in a few days once things calm down on the mirrors.

Stage

  • landed exclusion for contrib dir
  • tagged Bootstrap with RELEASE_AUTOMATION_M15_1
  • landed master.cfg update to use new tag
  • reconfig-ed production-1.9-master
  • triggered Stage step from the waterfall.
  • failed out when /builds was full; restarted after removing
/data/cltbld/firefox-3.0.16
/builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend/
/builds/3.0.15-3.5.5-major-update/patcher/temp/firefox/3.0.15/ftp/*
/builds/3.0.16-3.5.6-major-update/snippets/patcher/temp/firefox/3.0.16/ftp/*
  • No problems

Sign Installers

  • moved stage-{unsigned,signed} to stage-{unsigned,signed}-bad_rsync_for_analysis

On Stage complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.16/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.16/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

And merge in the solaris builds

rsync -av /pub/mozilla.org/firefox/nightly/3.0.16-candidates/build3/contrib/solaris_* contrib/

Update Bouncer

Changed the file locations from .../3.0.16/... to .../3.0.16-real/...

Push to mirrors

Make sure that you use ffxbld for 'push to mirrors', rather than cltbld (this is a change, but a desired one).

  • push the stage-merged directory to the releases area. This can take a while, and can take even longer if there are contrib builds (solaris) in the subdirs already. For this reason, I sync the win32 builds first, and then perform a general rsync to catch everything after. This allows mirrors to start picking up the win32 builds (our most used platform) a little quicker:
# on stage
mkdir -p /home/ftp/pub/firefox/releases/3.0.16-real/win32
rsync -av /data/cltbld/firefox-3.0.16/stage-merged/win32/ /home/ftp/pub/firefox/releases/3.0.16-real/win32/
rsync -av /data/cltbld/firefox-3.0.16/stage-merged/ /home/ftp/pub/firefox/releases/3.0.16-real/
  • edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.16-real) and remove the aborted release (3.0.16).

Final Verification

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz19-firefox-{win32,mac,linux}.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
    • No Problems
  • Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20091202-Firefox-3.0.16
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20091202-Firefox-3.0.16-test/$a");'

Publish Updates to Release Channel

Backup snip was run above. Enable release snippets for minor update

time ~/bin/pushsnip 20091202-Firefox-3.0.16
real	2m52.488s
user	0m0.178s
sys	0m8.499s

Then again, waiting for go on MU

time ~/bin/pushsnip 20091202-Firefox-3.0.16-3.5.6-MU
real	0m10.123s
user	0m0.017s
sys	0m0.570s
time ~/bin/pushsnip 20091202-Firefox-3.0.16-3.5.6-MU-beta
real	0m16.796s
user	0m0.024s
sys	0m0.580s

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.16-real latest-3.0

Release matching XULRunner build

XULRunner builds are produced by the automation, this is to groom them and push to the mirrors.

# cltbld@stage
VERSION=1.9.0.16
FX_VERSION=3.0.16
BUILD=3

mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp -pv ~/firefox-${FX_VERSION}/stage-merged/source/firefox-${FX_VERSION}-source.tar.bz2 \
  source/xulrunner-${VERSION}-source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/

Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.0.16, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:

cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp ~/firefox-${FX_VERSION}/stage-merged/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

If all is fine then

rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

(Mirror updating can take a bit, 10-15 mins this time)

Ping mfinkle or Mossop to update the links in these two wiki pages for the version change