Releases/Firefox 3.0.6/BuildNotes
From MozillaWiki
< Releases | Firefox 3.0.6
Contents
- 1 Build Engineers
- 2 Bonsai queries
- 3 Tags
- 4 Build data
- 5 Notes
- 5.1 Build 1
- 5.1.1 Tag
- 5.1.2 Source
- 5.1.3 Build & Repack
- 5.1.4 Sign
- 5.1.5 L10nVerify
- 5.1.6 Generate Updates
- 5.1.7 Publish Updates to Test Channels (betatest & releasetest)
- 5.1.8 Update Verify
- 5.1.9 Stage
- 5.1.10 Push updates to beta channel
- 5.1.11 Sign Installers
- 5.1.12 Update Bouncer
- 5.1.13 Push to mirrors
- 5.1.14 Final Verification
- 5.1.15 Publish Updates to Release Channel
- 5.1.16 Release
- 5.1 Build 1
Build Engineers
coop
Tracking release bug
Bonsai queries
Updated CVS Tags devmo page.
Tags
Build 1:
| Module | Branch | Tag | Pull date |
| cvsroot/mozilla | HEAD | FIREFOX_3_0_6_BUILD1 | 2009-01-15 15:58 PST |
| l10n/l10n | HEAD | FIREFOX_3_0_6_BUILD1 | 2009-01-16 16:36 PST |
Build data
| Type | Build ID | SHA1 | Push date | Build machine |
| [Windows installer/zip] | fx-win32-1.9-slave2 | |||
| [Mac compressed] | fx-mac-1.9-slave2 | |||
| [Linux compressed] | fx-linux-1.9-slave2 |
Notes
Build 1
Bootstrap Tag: RELEASE_AUTOMATION_M11
Setup before starting:
- updated master.cfg (no changes found)
- On linux slave (fx-linux-1.9-slave2):
- ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works
- removed:
- /builds/verify/firefox-20016-301-real-major
- /builds/verify/firefox-20018-304-real-major
- /builds/verify/firefox-3.0.5
- /builds/updates/firefox-3.0.5
- /builds/source/firefox-3.0.5
- /data/cltbld/firefox-3.0.5
- On the mac slave (fx-mac-1.9-slave2):
- removed nothing, as there was loads of space.
- On the windows slave (fx-win32-1.9-slave2):
- removed:
- /e/builds/buildbot/trunk-automation/twistd.log.???
- /e/builds/buildbot/trunk-automation/twistd.log.??
- /e/xr19rel/WINNT_5.2_Depend
- removed:
- Space on slaves before starting:
- fx-linux-1.9-slave2: 20G on /builds
- fx-mac-1.9-slave2: 20G on /
- fx-win32-1.9-slave2: 16.5G on d:, 5.50G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
- changed frequency of l10n_nightly_scheduler from [1,9,13,17,21] to [1] in master.cfg
- Kicked off automation:
buildbot sendchange --username=ccooper --master=localhost:9989 -m"Firefox 3.0.6build1 release" go
Tag
- Step Tag died right away:
ASSERT: Tag::Execute(): /builds/tags/FIREFOX_3_0_5_BUILD1/cvsroot already exists? at Bootstrap/Step/Tag.pm line 65.
- tagged new bootstrap.cfg as RELEASE_AUTOMATION_M11 and sent another sendchange
- second attempt: no problems
Source
- No problems
Build & Repack
- No problems
Sign
- Signing doc
- no problems.
L10nVerify
- Automated - no problems.
Generate Updates
- Automated - no problems.
Publish Updates to Test Channels (betatest & releasetest)
- Automated - no problems.
Update Verify
- linux - no problems.
- mac - no problems.
- win32 - similar issues to what we saw with 3.0.5. Ben/Ted rationalized it as follows:
[12:59pm] bhearsum: the chk files are supposed to be forced updates now [12:59pm] bhearsum: (which means the partial should contain the full file, not just a diff) [1:00pm] ted: for precisely this reason [1:00pm] ted: well, sort of [1:00pm] ted: the files differ between the zip and installer, so the partial was bogus [1:02pm] bhearsum: the MARs are generated against the zip contents ? [1:03pm] bhearsum: well, the chk files were certainly forced [1:03pm] ted: i think so, aren't they? [1:03pm] ted: isn't that the base reason we hit this problem? [1:03pm] bhearsum: i'm not entirely certain, to be honest, but it sounds right [1:03pm] bhearsum: yeah... [1:03pm] bhearsum: right [1:04pm] bhearsum: so the update verify test says they're different between the new installer, and the previous installer + mar [1:04pm] bhearsum: and the reason we force them is so teh MARs don't fail because of this [1:04pm] bhearsum: ergo, we get these sucky errors in the update verify log, but they won't cause issues [1:04pm] ted: yeah, so the chk files are still going to differ between the installer+partial and new installer [1:05pm] bhearsum: even in the complete mar [1:05pm] ted: but they should be identical in the new zip and old installer+partial [1:05pm] bhearsum: right
Stage
- Automated - no problems
Push updates to beta channel
# put snippets on beta $ sudo su - cltbld # make sure using latest version of scripts in mozilla/tools/release/bin/ $ cd bin $ cvs update . $ cd /opt/aus2/snippets/staging # note the required parameter must match what will be used with pushsnip below. $ time ~/bin/backupsnip 20090120-Firefox-3.0.6-beta Running /bin/tar cfvj /opt/aus2/snippets/backup/20090127-1-pre-20090120-Firefox-3.0.6-beta.tar.bz2 . real 35m49.357s user 0m37.752s sys 1m2.350s $ time ~/bin/pushsnip 20090120-Firefox-3.0.6-beta Running /usr/bin/rsync -PaO /opt/aus2/snippets/staging/20090120-Firefox-3.0.6-beta/ /opt/aus2/incoming/3 real 1m17.814s user 0m0.140s sys 0m5.725s
Sign Installers
Done manually using these installer-signing-instructions here.
- complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.6/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.6/stage-merged/ ~/bin/checksum-files .
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
Update Bouncer
- Done.
- Note for next release: Do not remove the Check Now bit on the Firefox-3.0.6 Products until well after the change to the rsync module (to prevent the likes of bug 464566)
Push to mirrors
- push the stage-merged directory to the releases area:
# on stage rsync -av /data/cltbld/firefox-3.0.6/stage-merged/ /home/ftp/pub/firefox/releases/3.0.6/
- edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.6) and remove the previous release (3.0.5).
Final Verification
- Verify that releasetest points to valid bouncer links:
# this can be run from anywhere cvs co mozilla/testing/release cd mozilla/testing/release/updates cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- lots of 301 errors for the isohunt mirror, got pulled by sentry, wasn't grabbing the Windows builds
- Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd /opt/aus2/snippets/staging/20090120-Firefox-3.0.6
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090120-Firefox-3.0.6-test/$a");'
$
Publish Updates to Release Channel
While waiting for the formal "go," ran:
-bash-3.2$ time ~/bin/backupsnip 20090120-Firefox-3.0.6 Running /bin/tar cfvj /opt/aus2/snippets/backup/20090203-1-pre-20090120-Firefox-3.0.6.tar.bz2 . real 46m45.374s user 0m38.342s sys 1m6.426s
After formal go, ran:
$ time ~/bin/pushsnip 20090120-Firefox-3.0.6 real 0m53.148s user 0m0.108s sys 0m4.112s
Release
- fix latest symlink
# cltbld@stage cd /home/ftp/pub/firefox/releases rm latest-3.0 && ln -s 3.0.6 latest-3.0
After release, there were issues with several mirrors. nthomas had to disable all updates while he and IT investigated.
From Sam:
For those curious, the issue we had that caused us to pull updates was that a db server was mistakenly(?) taken out of rotation for bouncer, which was causing partial updates to fail. IT is investigating why this happened. Leaving updates on the release channel for users would've meant all (or maybe just most) users receiving full updates -- a problem for users (large download) and a problem for mirrors (lots more bandwidth). We'll talk more about this at the post-mortem that I'll schedule tomorrow for either later this week or early next.
