Releases/Firefox 3.0.12/BuildNotes

Please be sure to use

UPDATE_PACKAGING_R8 in the bootstrap config, so that we start using SHA512 hashes for update snippets, bug 472790

- nthomas

Also please make sure that you use ffxbld for 'push to mirrors' and updating the latest-3.0 symlink , rather than cltbld (this is a change, but a desired one).

- bhearsum

Build Engineers

lsblakk
Tracking release bug

Bonsai queries

cvsroot

l10n

Notes

Build 1

Setup before starting:

On fx-win32-1.9-slave2:
- Deleted /e/xr19rel/WINNT_5.2_Depend
- Deleted /e/fx19l10nrel/WINNT_5.2_Depend
- Deleted /e/fx19rel/WINNT_5.2_Depend

On fx-linux-1.9-slave2:
- Deleted /builds/verify/firefox-3.0.11
- Deleted /builds/updates/firefox-3.0.11
- Deleted /builds/source/firefox-3.0.11
- Deleted /builds/data/cltbld/firefox-3.0.11
- Deleted /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
- Deleted /builds/tags/FIREFOX_3_0_6_BUILD1
- Deleted /builds/tags/FIREFOX_3_0_7_BUILD1
- Deleted /builds/tags/FIREFOX_3_0_7_BUILD2

On fx-mac-1.9-slave2:
- Deleted /builds/verify/firefox-3.0.11

Space free on slaves:
- fx-mac-1.9-slave2: 26G free on /
- fx-win32-1.9-slave2: 16.6G free on D, 14G free on E
- fx-linux-1.9-slave2: 22G free on /builds

Kicked off automation:

 buildbot sendchange --username=lsblakk --master=localhost:9989 -m"Firefox 3.0.12build1 release" go

Tag

No problems

Source

No problems

Build & Repack

No problems

Sign

Signing doc

No problems

L10nVerify

No problems

Generate Updates

No problems

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

Mac failed Update Verify because 3.0.11 throttling was still active from the major update.

Linux failed because pushing to hg would have created a new head.

Windows failed with "Step Updates died: shell call returned bad exit code: 1 at Bootstrap/Step.pm line 110."

After patching verify.sh to include ?force=1 to the aus url, all platforms succeeded on a second build, though win32 second build failed with normal binary difference errors.

Stage

No Problems

Partner Repacks

No problems

XULRunner

Had to kick off manually, the dependent scheduler did not fire after Stage step.

Builds were fine, msg sent to release-drivers to announce candidates.

Push updates to beta channel

  # cltbld@aus2-staging.m.o
  # make sure scripts are up to date
  cd bin
  cvs update
  cd /opt/aus2/snippets/staging/
  ~/bin/backupsnip 20090707-Firefox-3.0.12-beta
  ~/bin/pushsnip 20090707-Firefox-3.0.12-beta

Sign Installers

Done manually using these installer-signing-instructions here -- Only up to the STOP HERE

On Stage complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.12/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.12/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

Done. Note for next release: Do not remove the Check Now bit on the Firefox-3.0.12 Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

Make sure that you use ffxbld for 'push to mirrors' and updating the latest-3.0 symlink , rather than cltbld (this is a change, but a desired one).

push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.12/stage-merged/ /home/ftp/pub/firefox/releases/3.0.12/

edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.12) and remove the previous release (3.0.11).

Final Verification

hg clone http://hg.mozilla.org/build/tools
cd tools/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

- A few 404 errors for http://mozilla2.snt.utwente.nl so checked again when verify was done and the directory at http://mozilla2.snt.utwente.nl/firefox/releases/3.0.12/win32/zh-TW/ was populated

Before pushing final updates, verify that "release" and "releasetest" channel match:

# on aus2-staging
$ cd /opt/aus2/snippets/staging/20090707-Firefox-3.0.12
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090707-Firefox-3.0.12-test/$a");'
$

3.0.12 -> 3.5.1 Major Update Refresh

Snippet Generation

# ssh cltbld@prometheus-vm.build.mozilla.org
mkdir -p /builds/3.0.12-3.5.1-major-update/snippets
cd /builds/3.0.12-3.5.1-major-update/snippets

cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:ffxbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.5.1 3.0.12-3.5.1
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick verify

Check that releasetest = beta = release.

cd temp/firefox/3.0.12-3.5.1
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2 -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; system("diff -r -u $_ $a");'

Push to AUS

cd /builds/3.0.12-3.5.1-major-update/snippets/patcher/temp/firefox/3.0.12-3.5.1
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090721-Firefox-3.0.12-3.5.1-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20090721-Firefox-3.0.12-3.5.1-MU

Enable test snippets

# cltbld@aus2-staging
~/bin/backupsnip 20090625-Firefox-3.0.12-3.5.1-test
~/bin/pushsnip   20090625-Firefox-3.0.12-3.5.1-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-3012-351-major/
cd /builds/verify/firefox-3012-351-major/
hg clone http://hg.mozilla.org/build/tools

# get patch to trawl "Only in" directories
cd tools/release/common
# on mac & linux
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544
patch -p3 < patch

# everyone ...
cd ../updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

results everything the same as 3.0.11 -> 3.5.0 MU. aka, everything is OK

Publish Updates to Release Channel

updated AUS' config-dist.php and pushed because of the new MU at this time.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20090707-Firefox-3.0.12
~/bin/pushsnip 20090707-Firefox-3.0.12
~/bin/pushsnip 20090721-Firefox-3.0.12-3.5.1-MU

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.12 latest-3.0

Release matching XULRunner build

XULRunner builds are produced by the automation, this is to groom them and push to the mirrors.

# cltbld@stage
VERSION=1.9.0.12
FX_VERSION=3.0.12
BUILD=1

mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
  /home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp /home/ftp/pub/firefox/releases/${FX_VERSION}/source/firefox-${FX_VERSION}-source.tar.bz2 \
  source/xulrunner-${VERSION}-source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/

Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.0.12, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:

cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp /home/ftp/pub/firefox/releases/${FX_VERSION}/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

If all is fine then

rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/

(Mirror updating can take a bit, 10-15 mins this time)

Ping mfinkle or Mossop to update the links in these two wiki pages for the version change

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	GECKO190_20090706_RELBRANCH FIREFOX_3_0_12_BUILD1 FIREFOX_3_0_12_RELEASE
l10n/l10n	HEAD	GECKO190_20090706_RELBRANCH FIREFOX_3_0_12_BUILD1 FIREFOX_3_0_12_RELEASE