Build Engineers

rhelmer, joduinn

Bonsai queries

Was anything checked in after the "door closed"?

Last checkins:

• cvs 2008-03-03 11:15 PST
• l10n 2008-03-03 05:00 PST

Bugs

Filed version bump bug in bug 418926

Tags

Module	Tag	Pull date
cvsroot/mozilla	GECKO19b4_20080304_RELBRANCH	HEAD @ 2008-03-03 11:15 PST
	FIREFOX_3_0b4_RC1	GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST
	FIREFOX_3_0b4_RELEASE	GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:19 PST
l10n/l10n	GECKO19b4_20080304_RELBRANCH	HEAD @ 2008-03-03 05:00 PST
	FIREFOX_3_0b4_RC1	GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST
	FIREFOX_3_0b4_RELEASE	GECKO19b4_20080304_RELBRANCH @ 2008-03-03 16:27 PST

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer]				en-US:fx-win32-1.9-slave2 l10n:fx-win32-1.9-slave2
[Mac compressed]				en-US:fx-mac-1.9-slave2 l10n:fx-mac-1.9-slave2
[Linux compressed]				en-US:fx-linux-1.9-slave2 l10n:fx-linux-1.9-slave2

Notes

RC1

Bootstrap tag: RELEASE_AUTOMATION_M7_2

Setup before starting:

• Profile Guided Optimisation
  • need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
  • for the Windows slave
    • bring over the mk_add_options PROFILE_GEN_SCRIPT... line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
    • Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
    • bring over $ProfiledBuild = 1; from nightly tinder-config.pl to release one
    • Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
• used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
  • make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
  • restarted master
• compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
• On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
  • REMEMBER to use VNC when restarting mac slave
  • verify that both mac and linux have "xhost +" since last reboot
  • update tinderbox to RELEASE_AUTOMATION_M7_2
    • FIXME update should be automatic - bug 397554
    • cd /builds/tinderbox/mozilla/tools/tinderbox
    • cvs up -r RELEASE_AUTOMATION_M7_2
  • remove the contents of /builds/verify/*
    • FIXME should be automatically backed up or removed - bug 413178
  • restart buildbot
• On production-1.9-master
  • remove the contents of /builds/updates/
  • remove private and public areas
    • rm -rf /data/cltbld/firefox-*
      • FIXME should not be needed - bug 415970
    • rm -rf /builds/tags/* - did not do this, plenty of space
      • FIXME should not be needed - bug 415970
    • in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release
      • rm -rf /home/ftp/pub/firefox/nightly/2007*
      • FIXME should not be needed - bug 415970
• af was not tagged, so had to checkout a tagged locale so CVS would cache it
  • cvs co -r FIREFOX_3_0b4_RELEASE l10n/de

• kick off buildbot (run as cltbld):

buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC1" release

Prestage

• prestage removed /home/ftp/pub/firefox/nightly/3.0b3-candidates/ from the master, and that's needed later for l10n_verify. Manually synced the bits back from stage.m.o; patch on bug to fix this permanently

Tag

• nothing to do

Source

• nothing to do
• VERIFYFIX: confirmed that bug 409880 was truely fixed.

Build

• Mac was started from console not VNC, yielding "CFMessagePortCreateLocal failed" (security context problem in AliveTest).
  • Removed all tags from tinder/mozconfigs, restarted from VNC.
  • FIXME slaves should come up usable on boot - bug 417887

• Linux failed AliveTest (could not connect to X server)
  • Linux was rebooted and "xhost +" was not run.
  • Removed all tags from tinder/mozconfigs, restarted from VNC.
  • FIXME slaves should come up usable on boot - bug 417887

• SyncToStaging() failed
  • bug 420824 filed for immediate relief
  • commented out everything up to repack and resent change
  • FIXME related to bug 398148?

• (all platforms) Repack had leftover locale build override from 3.0b3 RC3
  • backed out in bug 418926

• Linux Repack did depend build, kernel was updated and bootstrap.cfg was not
  • manually clobbered last-built and re-ran linux

• (all platforms) Repack was re-run without cleaning logs, so had to copy hourly directory manually to candidates dir and re-push to staging.

• Linux Repack - rebuilt uk, zh-TW, zh-CN, because old builddir (derived from kernel version) was removed during run, which disrupted the previous run.

Sign

Signing doc

L10nVerify

Updates to Test Channel

• fallout from bug 410006 (prettyVersion feature); AUS dir "3 Beta 3" is created, not "3.0b3".
  • workaround patch in bug 418926, re-gen and re-push AUS config
  • FIXME permanent fix in bug 420947

• verify beta snippets versus betatest snippets
  • only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log

• verify beta snippets versus releasetest snippets
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'

• FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in bug 418926, recreated/repushed snippets

• ensure that new test snippets are the same as old
  • no changes expected

diff -r 20080304-Firefox-3.0b4-test.old 20080304-Firefox-3.0b4-test

• verify beta snippets versus betatest snippets
  • only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log

• verify beta snippets versus releasetest snippets
  • no changes expected
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080304-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080304-Firefox-3.0b4-test/$a");'

update Verify

• results uploaded to https://build.mozilla.org/logs/Firefox_3.0b4_RC1/
  • VERIFYFIX: need to manually remove bump, updateverify config bump only works for rc1 bug 415261

Stage

• ran out of disk space. Nagios warned us, but filled up remaining space faster then we could make more space.
  • FIXME: change nagios threshold to be higher, so we have more time to react.
  • FIXME should move updates off, and/or add more disk

Sign Installers

• NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/"
• Done manually using these installer-signing-instructions here
• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0b4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0b4/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Manually add entries to download.mozilla.org

Final Verification

• manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/
  • looked ok.
• VERIFYFIX verified that filename problems from 3.b2 have been fixed. bug 409394
  • verified ok.

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/

• rc1 stopped because of bug 421528

RC2

Bootstrap tag: RELEASE_AUTOMATION_M7_2

Setup before starting:

• Profile Guided Optimisation
  • need at least rev1.383 of tinderbox/build-seamonkey-util.pm when tagging tinderbox
  • for the Windows slave
    • bring over the mk_add_options PROFILE_GEN_SCRIPT... line from the nightly mozconfig to the release one, and the tweaks to CFLAGS & CXXFLAGS
    • Also pick up the new jemalloc line and removal of WIN32_REDIST_DIR
    • bring over $ProfiledBuild = 1; from nightly tinder-config.pl to release one
    • Bump fx-win32-1.9-slave2 up to 2GB of RAM, to match nightly VM
• used existing version bump bug to attached diff for bootstrap.cfg and master.cfg
  • make sure /builds/buildbot/trunk-automation-master/bootstrap.cfg is updated
  • restarted master
• compare nightly and release mozconfig/tinder-config.pl files, merge any desired changes
• On fx-mac-1.9-slave2, fx-win32-1.9-slave2, fx-linux-1.9-slave2
  • REMEMBER to use VNC when restarting mac slave
  • verify that both mac and linux have "xhost +" since last reboot
  • update tinderbox to RELEASE_AUTOMATION_M7_2
    • FIXME update should be automatic - bug 397554
    • cd /builds/tinderbox/mozilla/tools/tinderbox
    • cvs up -r RELEASE_AUTOMATION_M7_2
  • remove the contents of /builds/verify/*
    • FIXME should be automatically backed up or removed - bug 413178
  • restart buildbot
• On production-1.9-master
  • remove the contents of /builds/updates/
  • remove private and public areas
    • rm -rf /data/cltbld/firefox-*
      • FIXME should not be needed - bug 415970
    • rm -rf /builds/tags/* - did not do this, plenty of space
      • FIXME should not be needed - bug 415970
    • in /home/ftp/pub/firefox/nightly/, remove all except the 3.x-candidates directory for the previous release
      • rm -rf /home/ftp/pub/firefox/nightly/2007*
      • FIXME should not be needed - bug 415970

• Win32 Tinderbox was missing PGO support. Tinderbox must be copied on MSYS, symlinks do not work. Copied /e/builds/tinderbox/mozilla/tools/tinderbox/* to:

/e/fx19rel/
/e/fx19l10nrel/
/e/builds/tinderbox/fx19rel/
/e/builds/tinderbox/fx19l10nrel/

• disabled linux/mac builders in master.cfg

• kick off buildbot (run as cltbld):

buildbot sendchange --username=joduinn --master=localhost:9989 -m "Firefox 3.0 Beta 4 RC2" release

Need to update master.cfg and restart/reconfig the Buildbot master for a fix

Prestage

• automated, nothing to do

Tag

• automated, nothing to do

Source

• automated, nothing to do

Build

• only rebuilding win32, so symlinked linux/mac on stage:

cd /home/ftp/pub/firefox/nightly/3.0b4-candidates
ln -s `pwd`/rc1/*linux* `pwd`/rc2/
ln -s `pwd`/rc1/*mac* `pwd`/rc2/
# remove win32 ja-JP-mac
rm -v rc2/firefox-3.0b4.ja-JP-mac.win32*
# remove partials
rm -v rc2/*partial*
# on production-1.8-master, sync from stage
rsync -av cltbld@stage.mozilla.org:/home/ftp/pub/firefox/nightly/3.0b4-candidates/ /home/ftp/pub/firefox/nightly/3.0b4-candidates/

• turns out absolute links no longer work, at least unless you use the new root (/pub/mozilla.org) - bug 421783. Fix it up with

cd /home/ftp/pub/firefox/nightly/3.0b4-candidates/rc2/
find . -type l | xargs rm -v
ln -sv ../rc1/*linux* .
ln -sv ../rc1/*mac* .

Sign

Signing doc

L10nVerify

Updates to Test Channel

• verify beta snippets versus betatest snippets
  • only URL (download vs. stage) and appv ("3 Beta 3" vs. "3 Beta 3rc3")
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log

• verify beta snippets versus releasetest snippets
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'

• FIXME: PatcherConfig made beta like betatest not releasetest. Posted patch in bug 418926, recreated/repushed snippets

• ensure that new test snippets are the same as old
  • no changes expected

diff -r 20080308-Firefox-3.0b4-test 20080308-Firefox-3.0b4-test

• verify beta snippets versus betatest snippets
  • only URL (download vs. stage) and appv ("3 Beta 4" vs. "3 Beta 4rc1")
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/betatest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 2>&1 | tee /tmp/beta.log

• verify beta snippets versus releasetest snippets
  • no changes expected
  • FIXME automate this in bug 409449

cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4/
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");'

update Verify

• win32 FAILed due to:
  • binary diffs in .chk files; expected failure
  • TODO link bug
  • missing el and es-AR locales in 3.0b3
• linux FAILed due to missing el and es-AR locales in 3.0b3
• mac FAILed due to missing el and es-AR locales in 3.0b3

Stage

• ran out of disk space; cleared out 3.0b3-candidates, nightly/2008* files and restarted
  • FIXME: change nagios threshold to be higher, so we have more time to react.
  • FIXME should move updates off, and/or add more disk

Sign Installers

• NOTE: this was not done: "pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/"
• Done manually using these installer-signing-instructions here
• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0b4/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0b4/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• already done for rc1

Final Verification

• manually verified files under /data/cltbld/firefox-3.0b4/stage-merged/
  • looked ok.
• VERIFYFIX verified that filename problems from 3.b2 have been fixed. bug 409394
  • verified ok.

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0b4/stage-merged/ /home/ftp/pub/firefox/releases/3.0b4/

• TODO: note that in /data/cltbld, the "firefox-3.0b4" directory should probably be "firefox-3.0b4rc2". This would make it easier to confirm changes between RCs. This is done for some releases not all; should we always do this?

Updates to Beta Channel

In case you need to cancel/abort updates, the quickest way to do this is [1]

• on Mac build machines, run quick updateverify test for releastest channel

cd /builds/verify/firefox-3.0b4/updates
cat moz19-firefox-*.cfg > releasetest.cfg
sed -i.bak 's/betatest/releasetest/' releasetest.cfg
./verify.sh -t releasetest.cfg 2>&1 | tee releasetest.log
# upload result to build.m.o/logs

Should be all HTTP/200 OK results e.g.:

grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200

• On aus2-staging, ensure that releasetest channel contents match beta channel contents

cd /opt/aus2/snippets/staging/20080308-Firefox-3.0b4
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20080308-Firefox-3.0b4-test/$a");' 

No difs found - which is correct.

• Once QA and Website gives formal "go"

# put snippets on beta
# login to aus2-staging
$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ ~/bin/pushsnip 20080308-Firefox-3.0b4

• on Mac build machines, run quick updateverify test for beta channel

cd /builds/verify/firefox-3.0b4/updates
cat moz19-firefox-*.cfg > beta.cfg
sed -i.bak 's/betatest/beta/' beta.cfg
./verify.sh -t beta.cfg 2>&1 | tee beta.log
# upload result to build.m.o/logs

Should be all HTTP/200 OK results e.g.:

grep 'HTTP' releasetest.log  | grep -v Found | grep -v 200

Wall clock timings

Stopping live updates (just in case!)

If QA finds a problem with updates *after* we start pushing updates "live", the one-and-only-one way we know to stop updates from further spreading is by killing all updates!

See instructions at: https://intranet.mozilla.org/Build:Updates .