Firefox 3.0b2:BuildNotes

Build Engineers

bhearsum, joduinn, rhelmer

Bonsai queries

Was anything checked in after the "door closed"?

Last checkins:

• RC1
  • cvsroot
  • l10n

Bugs

Filed version bump bug in bug 407077

Tags

Module	Tag	Pull date
cvsroot/mozilla	GECKO19b2_20071210_RELBRANCH	HEAD @ 2007-12-10 11:15 PST
	FIREFOX_3_0b2_RELEASE	HEAD @ 2007-12-10 11:15 PST
	FIREFOX_3_0b2_RC1	HEAD @ 2007-12-10 11:15 PST
l10n/l10n	GECKO19b2_20071210_RELBRANCH	HEAD @ 2007-12-10 11:15 PST
	FIREFOX_3_0b2_RELEASE	HEAD @ 2007-12-10 11:15 PST
	FIREFOX_3_0b2_RC1	HEAD @ 2007-12-10 11:15 PST

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				en-US:fx-win32-1.9-slave2 l10n:fx-win32-1.9-slave2
[Mac compressed]				en-US:fx-mac-1.9-slave2 l10n:fx-mac-1.9-slave2
[Linux compressed]				en-US:fx-linux-1.9-slave2 l10n:fx-linux-1.9-slave2

Notes

Bootstrap tag: RELEASE_AUTOMATION_M6_3

Setup before starting:

• Switched buildslaves back to using production ssh keys, made sure all necessary hosts were accepted.
• Switched buildslaves' tinderbox-configs directory back to the main cvs repository ones.
• Checked out bootstrap configs to /builds/buildbot/trunk-automation-master/bootstrap-configs on production-trunk-automation. Symlinked bootstrap.cfg -> bootstrap-configs/fx-moz19-bootstrap.cfg.

• kick off buildbot (run as cltbld):

buildbot sendchange --username=bhearsum --master=localhost:9989 -m "Firefox 3.0 Beta 2 RC1" release

• Unit tests failed, had to patch Stage.pm (bug 407351) and restart.

• production-trunk-automation's hard drive (/) would not mount rw, was ro.
  • Aravind took a look and fixed that. (bug 407796). Restarted from Source step.

Tag

• Completed OK but died because the last BuildStep was not updated for production (it was using staging-trunk-automation). Ran this manually and restarted with Tag commented out:

cvs -d:ext:cltbld@cvs.mozilla.org:/l10n co -r FIREFOX_3_0b2_RELEASE l10n/be

Because of disk failures, we ran some extra sanity checks on the tag:

# check out Mozilla trunk
cvs -d cltbld@cvs.mozilla.org:/cvsroot co mozilla/client.mk
cd mozilla
make -f client.mk MOZ_CO_DATE='2007-12-10 11:15 PST' MOZ_CO_PROJECT=browser checkout
# only version bump changes expected
# this is ok - "cvs diff: tag GECKO19b2_20071210_RELBRANCH is not in file .cvsignore"
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH > trunk_vs_relbranch.diff
# no changes expected
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH -r FIREFOX_3_0b2_RELEASE > release_vs_relbranch.diff
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH -r FIREFOX_3_0b2_RC1 > rc1_vs_relbranch.diff

# check out l10n
cvs -d cltbld@cvs.mozilla.org:/l10n co -D '2007-12-10 11:15 PST' l10n
cd l10n
cvs -q diff -u -r FIREFOX_3_0b2_RELEASE > release_vs_trunk.diff
cvs -q diff -u -r FIREFOX_3_0b2_RC1 > release_vs_rc1.diff

Source

• Failed setting a few permissions -- source tarball appears to have been transferred OK

rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/.": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3/unsigned": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3/unsigned/windows-xpi": Operation not permitted (1)

• fixed this one by doing this on production-trunk-automation:

chmod g-s /home/ftp/pub/firefox/nightly/

sent 2321456865 bytes  received 5255860 bytes  5507012.37 bytes/sec
total size is 4669948286  speedup is 2.01
rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

Step Source died: ASSERT: SyncToStaging(): rsync failed
collect: Cannot write ./dflBALmOFp014842 (bfcommit, uid=500, gid=51): Read-only file system
queueup: cannot create queue file ./qflBALmOFp014842, euid=500, fd=-1, fp=0x0: Read-only file system

• Investigating this failure still.

• Netapp monuted hard drive seemed to get mounted read-only, not sure why. Restarted production-trunk-automation

• Moved ~cltbld/trunk-automation to /builds/trunk-automation, adjusted buildbot.tac file to reflect new directory, restarted.

Build

• Repack on OSX failed because Apache wasn't running on production-trunk-automation.
  • Started Apache, comment out everything but 'Repack (execute)' and onwards in the 'Build' Builders and re-ran OS X manually. Linux and Win32 should be fine.

• Build on win32 did not have WIN32_REDIST_DIR set, resulting in 407988. Landed patch in that bug and rebuilt.

• Tag, Source, Build steps commented out and process resumed.

• Mac had the wrong path to the ffxbld key, so uploadsymbols failed. rhelmer reran uploadsymbols and saved output to /builds/tinderbox/Fx-Mozilla1.9-Release/3.0b2rc1-symbols-rhelmer.log

Sign

Bit again by bug 407962. Explicitly pulled MOZILLA_1_9a3_RELEASE from mozilla/tools/update-packaging to avoid the bug (checked out by sign-release.pl in checkouts/tools/update-tools).

L10nVerify

• gave some false positive reports for locales we did not ship (lt and ka); this is because the l10nverify area is reused, and some leftover testing files were there. Need to file a bug to make automation error out or do something intelligent in this case.

Updates to Test Channel

• patches came out wrong due to use MOZILLA_1_9a8_RELEASE tools/update-packaging (patches contained both complete and partial data), backed down to MOZILLA_1_9a3_RELEASE and it worked ok.

• found typo in mac a2 config, fixed and regenerated snippets by hand:

# on production-trunk-automation
cd /builds/updates/firefox-3.0b2/
# moved old aus2 dir
mkdir patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo
mv patcher/temp/firefox/3.0b1-3.0b2/aus2* patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo/
# update patcher config
cd configs
cvs up
cd ../
# generated snippets:
cd patcher
./patcher2.pl --app=firefox --config=../config/moz19-branch-patcher2.cfg --create-patchinfo
# pushed to aus2
cd temp/firefox/3.0b1-3.0b2/
rsync -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071212-Firefox-3.0b2/
rsync -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071212-Firefox-3.0b2-test/
# on aus2-staging
# correct buildid
mv /opt/aus2/incoming/3/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/ /opt/aus2/incoming/3/Firefox/3.0a2/Darwin_Universal-gcc3/2007020618/
# verify
cd /opt/aus2/snippets/staging/
rm -rfv 20071212-Firefox-3.0b2-test/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/
diff -r 20071212-Firefox-3.0b2-test/Firefox/ /opt/aus2/incoming/3/Firefox/
# lists 1.5, 2.0 only
# remove prod snippet too
rm -rfv 20071212-Firefox-3.0b2/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/

• had to remove a2,a3,a4 updates due to bug 408610

# on production-trunk-automation
cd /builds/updates/firefox-3.0b2/
# moved old aus2 dir
mkdir patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo
mv patcher/temp/firefox/3.0b1-3.0b2/aus2* patcher/temp/firefox/3.0b1-3.0b2/old.bug408610/
# update patcher config
cd configs
cvs up
cd ../
# generated snippets:
cd patcher
./patcher2.pl --app=firefox --config=../config/moz19-branch-patcher2.cfg --create-patchinfo
# pushed to aus2
cd temp/firefox/3.0b1-3.0b2/
rsync -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071217-Firefox-3.0b2/
rsync -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071217-Firefox-3.0b2-test/
# on aus2-staging
cd /opt/aus2/snippets/staging
# verify the test snippets
$ diff -r 20071212-Firefox-3.0b2-test 20071217-Firefox-3.0b2-test
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a2
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a3
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a4
# verify the production snippets
$ diff -r 20071212-Firefox-3.0b2 20071217-Firefox-3.0b2
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a2
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a3
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a4
#force the currently betatest to now match what is newly generated
#ie remove the a2/3/4 snippets
$ cd /opt/aus2/incoming/3/Firefox
$ mkdir /tmp/snippets
$ mv 3.0a2 /tmp/snippets/
$ mv 3.0a3 /tmp/snippets/
$ mv 3.0a4 /tmp/snippets/

update Verify

• updateverify config bumper assumes that the version in the release dir and the version in the filename are the same; this is not the same for win32 and mac for alphas and betas but is for linux e.g. (note "3.0 Beta 1" versus "3.0b1"):
  • /firefox/releases/3.0b1/mac/en-US/Firefox 3.0 Beta 1.dmg
  • /firefox/releases/3.0b1/linux-i686/en-US/firefox-3.0b1.tar.bz2

• lt and ko locales were dropped, so updateverify configs had to be manually tweaked

• since updateverify does not ensure that the updates dir is removed, there was still a checkout on win32 from the test staging run. Needs bug filed.

• updateverify crashed initially on win32; used workaround described in https://bugzilla.mozilla.org/show_bug.cgi?id=401327#c11

Stage

• retagged automation to pick up bug 407351
• moved /data/cltbld/firefox-3.0b2/batch1 to /data/cltbld/firefox-3.0b1/batch1_missing_linux on production-trunk-automation and moved /data/cltbld/firefox-3.0b2 to /data/cltbld/firefox-3.0b2.bug407351 on stage.m.o
• reran Stage step

Sign Installers

Done manually.

• pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/ to keymaster

using these installer-signing-instructions here

• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0b2/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0b2/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

done manually.

Final Verification

done manually on irc

Push to mirrors

• push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0b2/stage-merged/ /home/ftp/pub/firefox/releases/3.0b2/

Updates to Beta Channel

• on Mac, run quick verify test
  • FIXME automate me

cd /builds/verify/updates
cat moz19-firefox-*.cfg > releasetest.cfg
sed -i.bak 's/betatest/releasetest/' releasetest.cfg
./verify.sh -t releasetest.cfg 2>&1 | tee releasetest.log
# upload result to build.m.o/logs

• Beta URL was pointing to the FTP not download; was corrected in the post-automation version (still in /mofo).
  • regenerated, pushed as 20071218-Firefox-3.0b2
    • verified that 20071218-Firefox-3.0b2-test matches 20071218-Firefox-3.0b2-test
    • verified that only diff between 20071218-Firefox-3.0b2 and 20071217-Firefox-3.0b2 was the URL change
  • FIXME sync this version to public CVS when the public CVS patches to bootstrap land

• Ensure that releasetest channel contents match beta channel contents
  • FIXME should do this verification earlier; could be done when snippets are pushed to staging area

cd /opt/aus2/snippets/staging/20071218-Firefox-3.0b2
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20071218-Firefox-3.0b2-test/$a");' 2>&1 | tee ~/fx-3.0b2.diff

The only difference in ~/fx-3.0b2.diff should be "3.0b2rc1" versus "3.0b2".

• Once QA and Website gives formal "go"

# put snippets on beta
$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ ~/bin/pushsnip 20071218-Firefox-3.0b2

• on Mac, run quick verify test
  • FIXME automate me

cd /builds/verify/updates
cat moz19-firefox-*.cfg > beta.cfg
sed -i.bak 's/betatest/beta/' beta.cfg
./verify.sh -t beta.cfg 2>&1 | tee beta.log
# upload result to build.m.o/logs