Firefox 3.0b2:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

bhearsum, joduinn, rhelmer

Bonsai queries

Was anything checked in after the "door closed"?

Last checkins:

Bugs

Filed version bump bug in bug 407077

Tags

Module Tag Pull date
cvsroot/mozilla GECKO19b2_20071210_RELBRANCH HEAD @ 2007-12-10 11:15 PST
FIREFOX_3_0b2_RELEASE HEAD @ 2007-12-10 11:15 PST
FIREFOX_3_0b2_RC1 HEAD @ 2007-12-10 11:15 PST
l10n/l10n GECKO19b2_20071210_RELBRANCH HEAD @ 2007-12-10 11:15 PST
FIREFOX_3_0b2_RELEASE HEAD @ 2007-12-10 11:15 PST
FIREFOX_3_0b2_RC1 HEAD @ 2007-12-10 11:15 PST

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] en-US:fx-win32-1.9-slave2 l10n:fx-win32-1.9-slave2
[Mac compressed] en-US:fx-mac-1.9-slave2 l10n:fx-mac-1.9-slave2
[Linux compressed] en-US:fx-linux-1.9-slave2 l10n:fx-linux-1.9-slave2

Notes

Bootstrap tag: RELEASE_AUTOMATION_M6_3

Setup before starting:

  • Switched buildslaves back to using production ssh keys, made sure all necessary hosts were accepted.
  • Switched buildslaves' tinderbox-configs directory back to the main cvs repository ones.
  • Checked out bootstrap configs to /builds/buildbot/trunk-automation-master/bootstrap-configs on production-trunk-automation. Symlinked bootstrap.cfg -> bootstrap-configs/fx-moz19-bootstrap.cfg.
  • kick off buildbot (run as cltbld):
buildbot sendchange --username=bhearsum --master=localhost:9989 -m "Firefox 3.0 Beta 2 RC1" release
  • Unit tests failed, had to patch Stage.pm bug 407351 and restart.
  • production-trunk-automation's hard drive (/) would not mount rw, was ro.
    • Aravind took a look and fixed that. bug 407796. Restarted from Source step.

Tag

  • Completed OK but died because the last BuildStep was not updated for production (it was using staging-trunk-automation). Ran this manually and restarted with Tag commented out:
cvs -d:ext:cltbld@cvs.mozilla.org:/l10n co -r FIREFOX_3_0b2_RELEASE l10n/be

Because of disk failures, we ran some extra sanity checks on the tag: 

# check out Mozilla trunk
cvs -d cltbld@cvs.mozilla.org:/cvsroot co mozilla/client.mk
cd mozilla
make -f client.mk MOZ_CO_DATE='2007-12-10 11:15 PST' MOZ_CO_PROJECT=browser checkout
# only version bump changes expected
# this is ok - "cvs diff: tag GECKO19b2_20071210_RELBRANCH is not in file .cvsignore"
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH > trunk_vs_relbranch.diff
# no changes expected
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH -r FIREFOX_3_0b2_RELEASE > release_vs_relbranch.diff
cvs -q diff -u -r GECKO19b2_20071210_RELBRANCH -r FIREFOX_3_0b2_RC1 > rc1_vs_relbranch.diff

# check out l10n
cvs -d cltbld@cvs.mozilla.org:/l10n co -D '2007-12-10 11:15 PST' l10n
cd l10n
cvs -q diff -u -r FIREFOX_3_0b2_RELEASE > release_vs_trunk.diff
cvs -q diff -u -r FIREFOX_3_0b2_RC1 > release_vs_rc1.diff

Source

  • Failed setting a few permissions -- source tarball appears to have been transferred OK
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/.": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3/unsigned": Operation not permitted (1)
rsync: failed to set permissions on "/home/ftp/pub/firefox/nightly/3.0b1-candidates/rc3/unsigned/windows-xpi": Operation not permitted (1)
  • fixed this one by doing this on production-trunk-automation:
chmod g-s /home/ftp/pub/firefox/nightly/

sent 2321456865 bytes  received 5255860 bytes  5507012.37 bytes/sec
total size is 4669948286  speedup is 2.01
rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]

Step Source died: ASSERT: SyncToStaging(): rsync failed
collect: Cannot write ./dflBALmOFp014842 (bfcommit, uid=500, gid=51): Read-only file system
queueup: cannot create queue file ./qflBALmOFp014842, euid=500, fd=-1, fp=0x0: Read-only file system
  • Investigating this failure still.
  • Netapp monuted hard drive seemed to get mounted read-only, not sure why. Restarted production-trunk-automation
  • Moved ~cltbld/trunk-automation to /builds/trunk-automation, adjusted buildbot.tac file to reflect new directory, restarted.

Build

  • Repack on OSX failed because Apache wasn't running on production-trunk-automation.
    • Started Apache, comment out everything but 'Repack (execute)' and onwards in the 'Build' Builders and re-ran OS X manually. Linux and Win32 should be fine.
      • Apache has been set to start on boot on production-trunk-automation
  • Build on win32 did not have WIN32_REDIST_DIR set, resulting in 407988. Landed patch in that bug and rebuilt.
  • Tag, Source, Build steps commented out and process resumed.
  • Mac had the wrong path to the ffxbld key, so uploadsymbols failed. rhelmer reran uploadsymbols and saved output to /builds/tinderbox/Fx-Mozilla1.9-Release/3.0b2rc1-symbols-rhelmer.log
    • It appears to have the right key now.

Sign

Bit again by bug 407962. Both bug 412006 and bug 409479 track what versions of what signing work is being used right. Explicitly pulled MOZILLA_1_9a3_RELEASE from mozilla/tools/update-packaging to avoid the bug (checked out by sign-release.pl in checkouts/tools/update-tools).

  • FIXME needs verify step
    • look in the log file for errors

L10nVerify

  • gave some false positive reports for locales we did not ship (lt and ka); this is because the l10nverify area is reused, and some leftover testing files were there. 409430

Updates to Test Channel

  • patches came out wrong due to use MOZILLA_1_9a8_RELEASE tools/update-packaging (patches contained both complete and partial data), backed down to MOZILLA_1_9a3_RELEASE and it worked ok.
  • found typo in mac a2 config, fixed and regenerated snippets by hand:
# on production-trunk-automation
cd /builds/updates/firefox-3.0b2/
# moved old aus2 dir
mkdir patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo
mv patcher/temp/firefox/3.0b1-3.0b2/aus2* patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo/
# update patcher config
cd configs
cvs up
cd ../
# generated snippets:
cd patcher
./patcher2.pl --app=firefox --config=../config/moz19-branch-patcher2.cfg --create-patchinfo
# pushed to aus2
cd temp/firefox/3.0b1-3.0b2/
rsync -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071212-Firefox-3.0b2/
rsync -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071212-Firefox-3.0b2-test/
# on aus2-staging
# correct buildid
mv /opt/aus2/incoming/3/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/ /opt/aus2/incoming/3/Firefox/3.0a2/Darwin_Universal-gcc3/2007020618/
# verify
cd /opt/aus2/snippets/staging/
rm -rfv 20071212-Firefox-3.0b2-test/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/
diff -r 20071212-Firefox-3.0b2-test/Firefox/ /opt/aus2/incoming/3/Firefox/
# lists 1.5, 2.0 only
# remove prod snippet too
rm -rfv 20071212-Firefox-3.0b2/Firefox/3.0a2/Darwin_Universal-gcc3/2006020618/

# on production-trunk-automation
cd /builds/updates/firefox-3.0b2/
# moved old aus2 dir
mkdir patcher/temp/firefox/3.0b1-3.0b2/old.mac_typo
mv patcher/temp/firefox/3.0b1-3.0b2/aus2* patcher/temp/firefox/3.0b1-3.0b2/old.bug408610/
# update patcher config
cd configs
cvs up
cd ../
# generated snippets:
cd patcher
./patcher2.pl --app=firefox --config=../config/moz19-branch-patcher2.cfg --create-patchinfo
# pushed to aus2
cd temp/firefox/3.0b1-3.0b2/
rsync -av aus2/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071217-Firefox-3.0b2/
rsync -av aus2.test/ aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20071217-Firefox-3.0b2-test/
# on aus2-staging
cd /opt/aus2/snippets/staging
# verify the test snippets
$ diff -r 20071212-Firefox-3.0b2-test 20071217-Firefox-3.0b2-test
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a2
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a3
Only in 20071212-Firefox-3.0b2-test/Firefox: 3.0a4
# verify the production snippets
$ diff -r 20071212-Firefox-3.0b2 20071217-Firefox-3.0b2
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a2
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a3
Only in 20071212-Firefox-3.0b2/Firefox: 3.0a4
#force the currently betatest to now match what is newly generated
#ie remove the a2/3/4 snippets
$ cd /opt/aus2/incoming/3/Firefox
$ mkdir /tmp/snippets
$ mv 3.0a2 /tmp/snippets/
$ mv 3.0a3 /tmp/snippets/
$ mv 3.0a4 /tmp/snippets/

update Verify

  • updateverify config bumper assumes that the version in the release dir and the version in the filename are the same; this is not the same for win32 and mac for alphas and betas but is for linux e.g. (note "3.0 Beta 1" versus "3.0b1"):
    • /firefox/releases/3.0b1/mac/en-US/Firefox 3.0 Beta 1.dmg
    • /firefox/releases/3.0b1/linux-i686/en-US/firefox-3.0b1.tar.bz2
    • bug 409394
  • lt and ka locales were dropped, so updateverify configs had to be manually tweaked. 409434
  • since updateverify does not ensure that the updates dir is removed, there was still a checkout on win32 from the test staging run. bug 409434
  • partial updates should use the correct format of the version string. bug 410006
  • partial updates were noticed to be larger than complete updates, which is wrong. Fixed and regenerated partial updates. bug 412000 is tracking automatically verifying this in future.

Stage

  • retagged automation to pick up bug 407351
  • moved /data/cltbld/firefox-3.0b2/batch1 to /data/cltbld/firefox-3.0b1/batch1_missing_linux on production-trunk-automation and moved /data/cltbld/firefox-3.0b2 to /data/cltbld/firefox-3.0b2.bug407351 on stage.m.o
  • reran Stage step

Sign Installers

Done manually.

  • pull stage:/data/cltbld/firefox-3.0b2/batch1/stage-signed/ to keymaster

using these installer-signing-instructions here

  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0b2/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0b2/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

done manually.

Final Verification

done manually on irc

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0b2/stage-merged/ /home/ftp/pub/firefox/releases/3.0b2/
  • discovered that the mac & win32 installer files on the mirrors had the wrong filename format ("Firefox Setup 3.0b2.exe" vs "Firefox Setup 3.0 Beta 2.exe"). This was caused by bug 409394 - see UpdateVerify section above.
    • we manually renamed all the mac & win32 installers, brought them back to keymaster, verified that the win32 authenticode was still ok, re-signed mac & win32 with GPG, pushed renamed files and the new GPG .asc files back up to mirrors.

Updates to Beta Channel

  • on Mac, run quick updateverify test 
cd /builds/verify/firefox-3.0b3/updates
cat moz19-firefox-*.cfg > releasetest.cfg
sed -i.bak 's/betatest/releasetest/' releasetest.cfg
./verify.sh -t releasetest.cfg 2>&1 | tee releasetest.log
# upload result to build.m.o/logs
  • Beta URL was pointing to the FTP not download; was corrected in the post-automation version (still in /mofo).
    • regenerated, pushed as 20071218-Firefox-3.0b2
      • verified that 20071218-Firefox-3.0b2-test matches 20071218-Firefox-3.0b2-test
      • verified that only diff between 20071218-Firefox-3.0b2 and 20071217-Firefox-3.0b2 was the URL change
    • FIXME sync this version to public CVS when the public CVS patches to bootstrap land
  • Ensure that releasetest channel contents match beta channel contents
    • Should do this verification earlier; could be done when snippets are pushed to staging area. bug 409449
cd /opt/aus2/snippets/staging/20071218-Firefox-3.0b2
find -type d -iregex '.*beta.*' | perl -nle '$a = $_; $a =~ s/beta/releasetest/; system("diff -r -u $_ ../20071218-Firefox-3.0b2-test/$a");' 2>&1 | tee ~/fx-3.0b2.diff

The only difference in ~/fx-3.0b2.diff should be "3.0b2rc1" versus "3.0b2". FIXME this is wrong ^ there should have been no diffs. disabled updates, generated 20071218.2-Firefox-3.0b2, pushed, enabled updates.


  • Once QA and Website gives formal "go"
# put snippets on beta
$ sudo su - cltbld
$ cd /opt/aus2/snippets/staging
$ ~/bin/pushsnip 20071218-Firefox-3.0b2

FIXME actually went live with 20071218.2-Firefox-3.0b2 as per above ^

  • on Mac, run quick verify test 
cd /builds/verify/updates
cat moz19-firefox-*.cfg > beta.cfg
sed -i.bak 's/betatest/beta/' beta.cfg
./verify.sh -t beta.cfg 2>&1 | tee beta.log
# upload result to build.m.o/logs

Free Software builds

Not doing Free Software builds for trunk. See bug#404464.

Wall clock timings

Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox_3.0b2:BuildNotes&oldid=83315"