Firefox 2.0.0.12:BuildNotes

Build Engineers

Version/config bump bug 414327

Bonsai queries

• RC1
  • http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=all&branch=MOZILLA_1_8_BRANCH&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-01-28+00%3A01+PST&maxdate=2008-01-28+16%3A00+PST&cvsroot=%2Fcvsroot
  • (l10n bonsai link goes here)
• RC2
  • (based on relbranch)
• RC3
  • (based on relbranch)

Tags

Module	Tag	Pull date
cvsroot/mozilla	RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1 FIREFOX_2_0_0_12_RELEASE	2008-01-28 16:00 PST
cvsroot/mozilla	RC2: FIREFOX_2_0_0_12_RC2 FIREFOX_2_0_0_12_RELEASE	2008-01-30 13:00 PST
mofo/talkback/fullsoft	RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1	2008-01-28 16:00 PST
l10n/l10n	RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1	2008-01-28 16:00 PST

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]	RC1:			production-pacifica-vm
[Mac compressed]	RC1:			bm-xserve05
[Linux compressed]	RC1:			production-prometheus-vm

Notes

RC1

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

• used existing version bump bug to attached diff for bootstrap.cfg
  • "cvs update" in /home/buildmaster/Automation/bootstrap-configs/ after checkin. Did a cvs stat first to work out if any local patches or if unexpected changes would be pulled in
  • restarted master
• ensure that machines have enough resources (bug 393274)
  • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
  • On bm-xserve05 remove the contents of /builds/verify/*
  • On build-console, make sure both private and public areas are synced to stage:
    • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
    • removed /home/ftp/pub/thunderbird/ after comparing to staging like above
    • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
      • expect new directories, but file contents should be the same
  • On build-console, remove private and public areas:
    • rm -rf /data/cltbld/firefox-*
    • rm -rf /builds/tags/* - did not do this, plenty of space
    • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
• kick off buildbot (run as cltbld):
  • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC1" release

Source

• permissions problems in /home/ftp/pub/firefox/nightly prevented rsync to stage
  • to resolve: chmod a-s /home/ftp/pub/firefox/nightly
  • need to investigate what's causing this

Build

• win32 build failed, CVS_RSH was not set in environment
  • buildbot slave was started from cmd.exe not cygwin
  • FIXME make this run on boot so we can't get it wrong :)
  • removed FIREFOX_2_0_0_12_* tags on tinder-config.pl/mozconfig and re-ran
• mac build failed
  • removed FIREFOX_2_0_0_12_* tags on tinder-config.pl/mozconfig and re-ran
  • intermittent failure seen on staging:

----------- Output from MozillaAliveTest ------------- 
 2008-01-28 22:14:33.998 firefox-bin[9432] CFLog (0): CFMessagePort: bootstrap_register(): failed 1100 (0x44c), port = 0x3d03, name = 'Processes-0.2228225'
 See /usr/include/servers/bootstrap_defs.h for the error codes.
 2008-01-28 22:14:33.999 firefox-bin[9432] CFLog (99): CFMessagePortCreateLocal(): failed to name Mach port (Processes-0.2228225)
 CFMessagePortCreateLocal failed (name = Processes-0.2228225 error = 0)
----------- End Output from MozillaAliveTest ---------

Sign

Signing doc

L10nVerify

• Automated verify ran. For linux & mac, worked fine, nothing to do. For win32, failed out with expected error.

Updates to Test Channels (betatest & releasetest)

• automated, nothing to do.

update Verify

• automated, nothing to do.

Stage

• automated, nothing to do.
• waiting for QA go/nogo.
• bug 414856 declared showstopper.

RC2

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

• used existing version bump bug to attached diff for bootstrap.cfg
• ensure that machines have enough resources (bug 393274)
  • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
  • On bm-xserve05 remove the contents of /builds/verify/*
  • On build-console, make sure both private and public areas are synced to stage:
    • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
    • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
      • expect new directories, but file contents should be the same
  • On build-console, remove private and public areas:
    • rm -rf /data/cltbld/firefox-*
    • rm -rf /builds/tags/* - did not do this, plenty of space
    • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
• kick off buildbot (run as cltbld):
  • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC1" release

Tag

• tag respin code determined relbranch incorrectly
  • fixed and landed in bug 414966

Source

• automated, nothing to do.

Build

• had to disable part of tag workaround bug 414966 so win32 and mac builds would work
• linux, mac builds handed over to QA
• unsigned win32 builds were waiting to be signed, when QA discovered that bug 413250 has another exploit path. QA declare "nogo". Stopped waiting to respin.

RC3

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

• used existing version bump bug to attached diff for bootstrap.cfg
• ensure that machines have enough resources (bug 393274)
  • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
  • On bm-xserve05 remove the contents of /builds/verify/*
  • On production-1.8-master, make sure both private and public areas are synced to stage:
    • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
    • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
      • expect new directories, but file contents should be the same
  • On production-1.8-master, remove private and public areas:
    • rm -rf /data/cltbld/firefox-*
    • rm -rf /builds/tags/* - did not do this, plenty of space
    • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
      • rm -rf /home/ftp/pub/firefox/nightly/2008*
      • rm -rf /home/ftp/pub/firefox/nightly/2.0.0.12-candidates
• kick off buildbot (run as cltbld):
  • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC3" release

Tag

• automated, nothing to do.

Source

• automated, nothing to do.

Build

• automated, nothing to do.

Sign

Signing doc

Updates to Test Channels (betatest & releasetest)

• Initially failed due to existing /builds/updates/firefox-2.0.0.12/patcher/mozilla, removed /builds/updates/firefox-2.0.0.12/ and restarted.

update Verify

• Linux and Mac update verify failed initially, config bumper code problem bug 415261
• Win32 should have failed, but warned instead. reopened bug 408157

Updates to Beta Channel

• beta updates getting served completes not partials bug 388524
  • fixed this bug, had to regenerate and re-push snippets

• verify beta snippets versus betatest snippets

cd /opt/aus2/snippets/staging/20080201-02-Firefox-2.0.0.12-beta/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080201-02-Firefox-2.0.0.12-test/$a");' 2>&1 | tee /tmp/beta.log

• QA declare "nogo". Stopped waiting to respin.

RC4

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

• used existing version bump bug to attached diff for bootstrap.cfg
• attach diff to update verify, remove current version due to bug 415261
• ensure that machines have enough resources
  • FIXME should be automatically backed up or removed - bug needed
• On production-prometheus-vm remove the contents of /builds/updates/
  • FIXME need to do this or updates will fail - bug needed
• On bm-xserve05, production-pacifica-vm, production-prometheus-vm remove the contents of /builds/verify/*
  • FIXME should be automatically backed up or removed - bug needed
• On stage, rename ~/firefox-2.0.0.12 to ~/firefox-2.0.0.12-rc3
  • FIXME should be automatically backed up or removed - bug needed
• On production-1.8-master, remove private and public areas:
  • FIXME should be automatically backed up or removed - bug needed
  • rm -rf /data/cltbld/firefox-*
  • rm -rf /builds/tags/* - did not do this, plenty of space
  • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
    • rm -rf /home/ftp/pub/firefox/nightly/2008*
    • rm -rf /home/ftp/pub/firefox/nightly/2.0.0.12-candidates
• kick off buildbot (run as cltbld):
  • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC4" release

Tag

• automated, nothing to do.

Source

• automated, nothing to do.

Build

• automated, nothing to do.

Sign

Signing doc

L10nVerify

• automated, nothing to do.

Updates to Test Channels (betatest & releasetest)

• PatcherConfig undid the DisableCompleteUpdates change in bug 388524, bug reopened.
  • landed fix and re-generated AUS config:

# backup current config
cd /builds/updates/firefox-2.0.0.12/patcher/temp/firefox/2.0.0.11-2.0.0.12
mkdir rc4.1
mv aus2.* rc4.1
# pull in fixed config
cd /builds/updates/firefox-2.0.0.12/config/
cvs up
# regenerate AUS config
cd /builds/updates/firefox-2.0.0.12/patcher/
./patcher2.pl --app=firefox --config=../config/moz18-branch-patcher2.cfg --create-patchinfo 2>&1 | tee patchinfo-rc4_2.log

• push new config to AUS

rsync -e ssh -av aus2.beta/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-beta/
rsync -e ssh -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-test/
rsync -e ssh -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12/ 

• verify beta snippets versus betatest snippets
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-beta/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080203-Firefox-2.0.0.12-test/$a");' 2>&1 | tee /tmp/beta.log

update Verify

• automated, nothing to do.
• FIXME - should verify all previous releases, not just the immediately previous bug 408453

Stage

• automated, nothing to do.

Updates to Beta Channel

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ cd /opt/aus2/snippets/staging
 $ sudo ~cltbld/bin/pushsnip 20080203-Firefox-2.0.0.12-beta

• FIXME pushsnip does backup before pushing; we could do this backup in the Updates->Push() method instead. needs bug filed

Sign Installers

• NOTE - build-console is no longer used from here on
• pull stage:/data/cltbld/firefox-2.0.0.12/batch1/stage-signed/ to signing machine
• follow installer-signing-instructions here
• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/firefox-2.0.0.12/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-2.0.0.12/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Release

• get the last formal "go" from QA/Dev/website/IT/release-drivers
• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-2.0.0.12/stage-merged/ /home/ftp/pub/firefox/releases/2.0.0.12/

• After pushing live, move forward the "latest" and "latest-2.0" symlinks:

 cd /home/ftp/pub/firefox/releases/
 rm latest
 ln -s 2.0.0.12 latest
 rm latest-2.0
 ln -s 2.0.0.12 latest-2.0

• Update stage:/etc/rsyncd-mozilla-current.exclude

Update Bouncer

• Add entries to download.mozilla.org

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080203-Firefox-2.0.0.12-test/$a");'

Enable update channel

• Wait for QA to finish testing update snippets on releasetest channel, then enable update snippets on release channel

 # on aus2-staging
 # put snippets on release/live channel
 $ cd /opt/aus2/snippets/staging
 $ sudo su - cltbld
 $ ~/bin/pushsnip 20080203-Firefox-2.0.0.12

Free Software builds

• Bump the BuildTag definition in the three tinder-config.pl, like this

• Done on production machines:
  • production-pacifica-vm (win32)
  • bm-xserve05 (macosx)
  • production-prometheus-vm (linux)

• Start builds:

cd /builds/tinderbox/Fx-Mozilla1.8-FS
./build-seamonkey.pl --once --depend --config-cvsup-dir `pwd`/tinderbox-configs/ 2>&1 | tee FIREFOX_2_0_0_12_FS.log

• Stage

# as cltbld@stage
mkdir -p ~/firefox-2.0.0.12-fs/batch1/prestage
cd ~/firefox-2.0.0.12-fs/batch1
# rsync from push dirs
rsync -av /home/ftp/pub/firefox/nightly/2008-02-07-09-mozilla1.8-fs/ ./prestage/
rsync -av prestage/ prestage-trimmed/
rm -v prestage-trimmed/*.zip
rsync -av prestage-trimmed/ stage/
cd stage
~/bin/groom-files --long="2.0.0.12.fs" .
chown -R cltbld:firefox * .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ../
rsync -av stage/ /home/ftp/pub/firefox/releases/2.0.0.12/contrib/free-software/