- 1 Build Engineers
- 2 Bonsai queries
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build
- 5.1.4 Repack
- 5.1.5 Sign
- 5.1.6 L10nVerify
- 5.1.7 Generate Updates
- 5.1.8 Publish Updates to Test Channels (betatest & releasetest)
- 5.1.9 Update Verify
- 5.1.10 Stage
- 5.1.11 Push Updates to Beta Channels
- 5.1.12 Sign Installers
- 5.1.13 Update Bouncer
- 5.1.14 Push to mirrors
- 5.1.15 Final Verification
- 5.1.16 Publish Updates to Release Channel
- 5.1.17 Release
- 5.1 Build 1
Version/config bump bug
Updated CVS Tags devmo page.
|cvsroot/mozilla||HEAD||FIREFOX_3_0_1_BUILD1||2008-07-02 01:52 PDT|
|l10n/l10n||HEAD||FIREFOX_3_0_1_BUILD1||2008-07-02 01:52 PDT|
|Type||Build ID||SHA1||Push date||Build machine|
Bootstrap Tag: RELEASE_AUTOMATION_M10
Setup before starting:
- Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
- On the slaves:
- verify that linux has 'DISPLAY=:0 xhost +' set
- remove /builds/verify/firefox-3.0rc3 and firefox-3.0rc2
- update tinderbox to RELEASE_AUTOMATION_M10
- For win32:
cd /e/builds/tinderbox/mozilla/tools/tinderbox cvs up -r RELEASE_AUTOMATION_M10 for dir in fx19rel fx19l10nrel xr19rel; do cd /e/$dir; for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .; done cp -pv post-mozilla-rel.pl post-mozilla.pl done
- On the linux slave, removed:
- For some reason bm-xserve11 was attached to the buildbot master.
- Disconnected it, and reconnected the proper slave (bm-xserve10/fx-mac-1.9-slave2
- Space on slaves before starting:
- fx-linux-1.9-slave2: 29G on /builds
- fx-mac-1.9-slave2: 24G on /
- fx-win32-1.9-slave2: 17G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
- Disabled everything after Updates::Push in master.cfg (local modification) because additional patcherconfig/update work needs to be done after Updates::execute but before updates::push.
- Gulped, and kicked off automation:
buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.1build1 release" gogogo
Automated, no problems.
Automated, no problems
- TinderConfig failed with:
Step TinderConfig died: ASSERT: no replacement found for appVersion at Bootstrap/Config.pm line 364, <INFILE> line 287.
Not sure why this happened ('version' should be used when appVersion isn't present, I think. Maybe it's not when oldAppVersion is present?). Added it to the config, checked in, tagged, restarted builds.
- TinderConfig failed spuriously because en-US configs were already tagged. Commented out TinderConfig from master.cfg and restarted builds.
- Build failed out on Mac - /usr/bin/python was pointing to python2.3 - repointed it to /tools/python/bin/python (python 2.5). not sure how this happened, or how it worked for 3.0rc1/2/3. maybe related to above comments about bm-xserve11.
- Build failed out on win32 due to disk space. Deleted some trees on e: and restarted. (10G free @ time of restart)
- Mac repack was done incorrectly because of the TinderConfig problem mentioned above. Only the en-US tinder-config for mac and win32 was bumped. TinderConfig was manually re-run on win32 and mac. This was done before win32 started repacking (therefore, it did the right thing the first time), but after mac had completed its full repack. All of the mac repacks were deleted from stage, and Repack::Excute and ::Push were re-run. This produced the correct builds.
- No other problems
Mac snippets (3.0rc3 -> 3.0.1)
- Ran PatcherConfig & Updates::Execute to get Mac snippets.
- Didn't get a 'aus2.beta' directory (beta + release channel snippets were in the same directory).
- Fixed that by adding 'beta-dir' to <current-update> stanza in the patcher config
- Re-ran Updates::Execute - got aus2.beta this time
- Did a spot check, things looked OK.
Linux & win32 snippets (3.0rc2 -> 3.0.1)
- Updated patcher config to get linux & win32 partials/snippets (from: 3.0rc2, to: 3.0.1, comment out rc3 release stanza, comment out mac from rc2, comment out 3.0rc2 to 3.0rc3 past-update line)
- Commented out 'clean update area' so we don't clobber the mac snippets.
- backup mac snippets:
cp -R 3.0rc3-3.0.1/ 3.0rc3-3.0.1.rc3.mac cp -R 3.0.rc3-3.0.1/ ~/3.0rc3-3.0.1.rc3.mac
- Re-ran Updates::Execute to generate linux and win32 partials/snippets
- Got the following at the bottom of the log:
Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2.test: No such file or directory at Bootstrap/Step/Updates.pm line 110 Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2: No such file or directory at Bootstrap/Step/Updates.pm line 119
- This is because we're generating snippets for a release that is NOT the 'version' as set in bootstrap.cfg. Manual spotcheck below to make sure everything is ok.
- Manual spotchecks:
# in aus2.test: look for non-test channels snippets in test channel dir find . -mindepth 6 -type d | grep -v test # expect no output # the same thing for beta, and release channel # in aus2.beta find . -mindepth 6 -type d | grep -v beta # expect no output # in aus2 find . -mindepth 6 -type d | grep -v release # expect no output
- Inspected a few snippets from various places - they look OK.
- Re-checked-in automatically generated patcher config, kept 'beta-dir beta' line.
Merged snippets + partials into one directory with the following commands:
mkdir 3.0rc3-3.0.1 rsync -av 3.0rc3-3.0.1.rc3.mac/ 3.0rc3-3.0.1/ rsync -av 3.0rc2-3.0.1/ 3.0rc3-3.0.1/
Restarted automation at Updates::Push
Automation continued normally.
Publish Updates to Test Channels (betatest & releasetest)
- Linux passed.
- Mac had expected failure of gu-IN (because we stopped shipping it on Mac at rc1)
- Win32 had expected failures about binary files
Push Updates to Beta Channels
-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1-beta Running /bin/tar cfvj /opt/aus2/snippets/backup/20080703-4-pre-20080703-Firefox-3.0.1-beta.tar.bz2 . real 28m44.170s user 0m22.740s sys 0m33.067s -bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1-beta sent 737264 bytes received 44534 bytes 10354.94 bytes/sec total size is 588339 speedup is 0.75 real 1m15.405s user 0m0.107s sys 0m3.612s
Done manually using these installer-signing-instructions here
- complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.1/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.1/stage-merged/ ~/bin/checksum-files .
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
- Manually added entries to download.mozilla.org/admin.php
- Linux/win32 partials were added under the 'Firefox-3.0.1-Partial-3.0rc2' product with filenames like:
- Mac partials use 'Firefox-3.0.1-Partial-3.0rc3'
- Had to do this because we only shipped rc3 on Mac. Stage step renamed partials to 3.0rc3-3.0.1 but snippets point to, eg. http://download.mozilla.org/?product=firefox-3.0.1-partial-3.0rc2&os=win&lang=af
Push to mirrors
- push the stage-merged directory to the releases area:
# on stage rsync -av /data/cltbld/firefox-3.0.1/stage-merged/ /home/ftp/pub/firefox/releases/3.0.1/
- Verify that releasetest points to valid bouncer links:
# this can be run from anywhere cvs co mozilla/testing/release cd mozilla/testing/release/updates cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- Two 404s caused by mirrors in a half-synced state
- One 500 error, seems transient
- Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging cd /opt/aus2/snippets/staging/20080703-Firefox-3.0.1 find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080703-Firefox-3.0.1-test/$a");'
Publish Updates to Release Channel
-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1 Running /bin/tar cfvj /opt/aus2/snippets/backup/20080716-2-pre-20080703-Firefox-3.0.1.tar.bz2 . real 30m39.986s user 0m23.497s sys 0m34.947s -bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1
# on stage as cltbld cd /home/ftp/pub/firefox/releases rm latest ln -s latest-3.0 latest rm latest-3.0 ln -s 3.0.1 latest-3.0