Firefox 2.0.0.12:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

Version/config bump bug 414327

Bonsai queries

Tags

Module Tag Pull date
cvsroot/mozilla RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1 FIREFOX_2_0_0_12_RELEASE 2008-01-28 16:00 PST
cvsroot/mozilla RC2: FIREFOX_2_0_0_12_RC2 FIREFOX_2_0_0_12_RELEASE 2008-01-30 13:00 PST
mofo/talkback/fullsoft RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1 2008-01-28 16:00 PST
l10n/l10n RC1: GECKO181_20080128_RELBRANCH FIREFOX_2_0_0_12_RC1 2008-01-28 16:00 PST

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] RC1: production-pacifica-vm
[Mac compressed] RC1: bm-xserve05
[Linux compressed] RC1: production-prometheus-vm

Notes

RC1

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

  • used existing version bump bug to attached diff for bootstrap.cfg
    • "cvs update" in /home/buildmaster/Automation/bootstrap-configs/ after checkin. Did a cvs stat first to work out if any local patches or if unexpected changes would be pulled in
    • restarted master
  • ensure that machines have enough resources (bug 393274)
    • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
    • On bm-xserve05 remove the contents of /builds/verify/*
    • On build-console, make sure both private and public areas are synced to stage:
      • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
      • removed /home/ftp/pub/thunderbird/ after comparing to staging like above
      • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
        • expect new directories, but file contents should be the same
    • On build-console, remove private and public areas:
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
  • kick off buildbot (run as cltbld):
    • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC1" release

Tag

Source

  • permissions problems in /home/ftp/pub/firefox/nightly prevented rsync to stage
    • to resolve: chmod a-s /home/ftp/pub/firefox/nightly
    • need to investigate what's causing this

Build

  • win32 build failed, CVS_RSH was not set in environment
    • buildbot slave was started from cmd.exe not cygwin
    • FIXME make this run on boot so we can't get it wrong :)
    • removed FIREFOX_2_0_0_12_* tags on tinder-config.pl/mozconfig and re-ran
  • mac build failed
    • removed FIREFOX_2_0_0_12_* tags on tinder-config.pl/mozconfig and re-ran
    • intermittent failure seen on staging:
----------- Output from MozillaAliveTest ------------- 
 2008-01-28 22:14:33.998 firefox-bin[9432] CFLog (0): CFMessagePort: bootstrap_register(): failed 1100 (0x44c), port = 0x3d03, name = 'Processes-0.2228225'
 See /usr/include/servers/bootstrap_defs.h for the error codes.
 2008-01-28 22:14:33.999 firefox-bin[9432] CFLog (99): CFMessagePortCreateLocal(): failed to name Mach port (Processes-0.2228225)
 CFMessagePortCreateLocal failed (name = Processes-0.2228225 error = 0)
----------- End Output from MozillaAliveTest ---------

Sign

Signing doc

L10nVerify

  • Automated verify ran. For linux & mac, worked fine, nothing to do. For win32, failed out with expected error.

Updates to Test Channels (betatest & releasetest)

  • automated, nothing to do.

update Verify

  • automated, nothing to do.

Stage

  • automated, nothing to do.
  • waiting for QA go/nogo.
  • bug 414856 declared showstopper.

RC2

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

  • used existing version bump bug to attached diff for bootstrap.cfg
  • ensure that machines have enough resources (bug 393274)
    • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
    • On bm-xserve05 remove the contents of /builds/verify/*
    • On build-console, make sure both private and public areas are synced to stage:
      • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
      • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
        • expect new directories, but file contents should be the same
    • On build-console, remove private and public areas:
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
  • kick off buildbot (run as cltbld):
    • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC1" release

Tag

  • tag respin code determined relbranch incorrectly

Source

  • automated, nothing to do.

Build

  • had to disable part of tag workaround bug 414966 so win32 and mac builds would work
  • linux, mac builds handed over to QA
  • unsigned win32 builds were waiting to be signed, when QA discovered that bug 413250 has another exploit path. QA declare "nogo". Stopped waiting to respin.



RC3

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

  • used existing version bump bug to attached diff for bootstrap.cfg
  • ensure that machines have enough resources (bug 393274)
    • On production-prometheus-vm remove the contents of /builds/updates/* - didn't need to do this, plenty of space
    • On bm-xserve05 remove the contents of /builds/verify/*
    • On production-1.8-master, make sure both private and public areas are synced to stage:
      • rsync -nav /home/ftp/pub/firefox/nightly/2.0.0.11-candidates/ stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.11-candidates/
      • rsync -nav /data/cltbld/firefox-2.0.0.11/ stage:/data/cltbld/firefox-2.0.0.11/
        • expect new directories, but file contents should be the same
    • On production-1.8-master, remove private and public areas:
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
        • rm -rf /home/ftp/pub/firefox/nightly/2008*
        • rm -rf /home/ftp/pub/firefox/nightly/2.0.0.12-candidates
  • kick off buildbot (run as cltbld):
    • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC3" release

Tag

  • automated, nothing to do.

Source

  • automated, nothing to do.

Build

  • automated, nothing to do.

Sign

Signing doc

L10nVerify

Updates to Test Channels (betatest & releasetest)

  • Initially failed due to existing /builds/updates/firefox-2.0.0.12/patcher/mozilla, removed /builds/updates/firefox-2.0.0.12/ and restarted.

update Verify

  • Linux and Mac update verify failed initially, config bumper code problem bug 415261
  • Win32 should have failed, but warned instead. reopened bug 408157

Stage

Updates to Beta Channel

  • beta updates getting served completes not partials bug 388524
    • fixed this bug, had to regenerate and re-push snippets
  • verify beta snippets versus betatest snippets
cd /opt/aus2/snippets/staging/20080201-02-Firefox-2.0.0.12-beta/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080201-02-Firefox-2.0.0.12-test/$a");' 2>&1 | tee /tmp/beta.log
  • QA declare "nogo". Stopped waiting to respin.

RC4

Bootstrap tag: RELEASE_AUTOMATION_M7

Setup before starting:

  • used existing version bump bug to attached diff for bootstrap.cfg
  • attach diff to update verify, remove current version due to bug 415261
  • ensure that machines have enough resources
    • FIXME should be automatically backed up or removed - bug needed
  • On production-prometheus-vm remove the contents of /builds/updates/
    • FIXME need to do this or updates will fail - bug needed
  • On bm-xserve05, production-pacifica-vm, production-prometheus-vm remove the contents of /builds/verify/*
    • FIXME should be automatically backed up or removed - bug needed
  • On stage, rename ~/firefox-2.0.0.12 to ~/firefox-2.0.0.12-rc3
    • FIXME should be automatically backed up or removed - bug needed
  • On production-1.8-master, remove private and public areas:
    • FIXME should be automatically backed up or removed - bug needed
    • rm -rf /data/cltbld/firefox-*
    • rm -rf /builds/tags/* - did not do this, plenty of space
    • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
      • rm -rf /home/ftp/pub/firefox/nightly/2008*
      • rm -rf /home/ftp/pub/firefox/nightly/2.0.0.12-candidates
  • kick off buildbot (run as cltbld):
    • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC4" release

Tag

  • automated, nothing to do.

Source

  • automated, nothing to do.

Build

  • automated, nothing to do.

Sign

Signing doc

L10nVerify

  • automated, nothing to do.

Updates to Test Channels (betatest & releasetest)

  • PatcherConfig undid the DisableCompleteUpdates change in bug 388524, bug reopened.
    • landed fix and re-generated AUS config:
# backup current config
cd /builds/updates/firefox-2.0.0.12/patcher/temp/firefox/2.0.0.11-2.0.0.12
mkdir rc4.1
mv aus2.* rc4.1
# pull in fixed config
cd /builds/updates/firefox-2.0.0.12/config/
cvs up
# regenerate AUS config
cd /builds/updates/firefox-2.0.0.12/patcher/
./patcher2.pl --app=firefox --config=../config/moz18-branch-patcher2.cfg --create-patchinfo 2>&1 | tee patchinfo-rc4_2.log
  • push new config to AUS
rsync -e ssh -av aus2.beta/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-beta/
rsync -e ssh -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-test/
rsync -e ssh -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12/ 
  • verify beta snippets versus betatest snippets
cd /opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12-beta/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080203-Firefox-2.0.0.12-test/$a");' 2>&1 | tee /tmp/beta.log

update Verify

  • automated, nothing to do.
  • FIXME - should verify all previous releases, not just the immediately previous bug 408453

Stage

  • automated, nothing to do.

Updates to Beta Channel

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ cd /opt/aus2/snippets/staging
 $ sudo ~cltbld/bin/pushsnip 20080203-Firefox-2.0.0.12-beta
  • FIXME pushsnip does backup before pushing; we could do this backup in the Updates->Push() method instead. needs bug filed

Sign Installers

  • NOTE - build-console is no longer used from here on
  • pull stage:/data/cltbld/firefox-2.0.0.12/batch1/stage-signed/ to signing machine
  • follow installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-2.0.0.12/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-2.0.0.12/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Release

  • get the last formal "go" from QA/Dev/website/IT/release-drivers
  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-2.0.0.12/stage-merged/ /home/ftp/pub/firefox/releases/2.0.0.12/
  • After pushing live, move forward the "latest" and "latest-2.0" symlinks:
 cd /home/ftp/pub/firefox/releases/
 rm latest
 ln -s 2.0.0.12 latest
 rm latest-2.0
 ln -s 2.0.0.12 latest-2.0
  • Update stage:/etc/rsyncd-mozilla-current.exclude

Update Bouncer

  • Add entries to download.mozilla.org

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080203-Firefox-2.0.0.12
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080203-Firefox-2.0.0.12-test/$a");'

Enable update channel

  • Wait for QA to finish testing update snippets on releasetest channel, then enable update snippets on release channel
 # on aus2-staging
 # put snippets on release/live channel
 $ cd /opt/aus2/snippets/staging
 $ sudo su - cltbld
 $ ~/bin/pushsnip 20080203-Firefox-2.0.0.12

Free Software builds

  • Bump the BuildTag definition in the three tinder-config.pl, like this
  • Done on production machines:
    • production-pacifica-vm (win32)
    • bm-xserve05 (macosx)
    • production-prometheus-vm (linux)
  • Start builds:
cd /builds/tinderbox/Fx-Mozilla1.8-FS
./build-seamonkey.pl --once --depend --config-cvsup-dir `pwd`/tinderbox-configs/ 2>&1 | tee FIREFOX_2_0_0_12_FS.log
  • Stage
# as cltbld@stage
mkdir -p ~/firefox-2.0.0.12-fs/batch1/prestage
cd ~/firefox-2.0.0.12-fs/batch1
# rsync from push dirs
rsync -av /home/ftp/pub/firefox/nightly/2008-02-07-09-mozilla1.8-fs/ ./prestage/
rsync -av prestage/ prestage-trimmed/
rm -v prestage-trimmed/*.zip
rsync -av prestage-trimmed/ stage/
cd stage
~/bin/groom-files --long="2.0.0.12.fs" .
chown -R cltbld:firefox * .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ../
rsync -av stage/ /home/ftp/pub/firefox/releases/2.0.0.12/contrib/free-software/

Wall clock timings