Releases/Firefox 3.0.6/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

Tracking release bug

Bonsai queries

last cvs checkin

last l10n checkin

Updated CVS Tags devmo page.


Build 1:

Module Branch Tag Pull date
cvsroot/mozilla HEAD FIREFOX_3_0_6_BUILD1 2009-01-15 15:58 PST
l10n/l10n HEAD FIREFOX_3_0_6_BUILD1 2009-01-16 16:36 PST

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] fx-win32-1.9-slave2
[Mac compressed] fx-mac-1.9-slave2
[Linux compressed] fx-linux-1.9-slave2


Build 1


Setup before starting:

  • updated master.cfg (no changes found)
  • On linux slave (fx-linux-1.9-slave2):
    • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works
    • removed:
      • /builds/verify/firefox-20016-301-real-major
      • /builds/verify/firefox-20018-304-real-major
      • /builds/verify/firefox-3.0.5
      • /builds/updates/firefox-3.0.5
      • /builds/source/firefox-3.0.5
      • /data/cltbld/firefox-3.0.5
  • On the mac slave (fx-mac-1.9-slave2):
    • removed nothing, as there was loads of space.
  • On the windows slave (fx-win32-1.9-slave2):
    • removed:
      • /e/builds/buildbot/trunk-automation/twistd.log.???
      • /e/builds/buildbot/trunk-automation/twistd.log.??
      • /e/xr19rel/WINNT_5.2_Depend
  • Space on slaves before starting:
    • fx-linux-1.9-slave2: 20G on /builds
    • fx-mac-1.9-slave2: 20G on /
    • fx-win32-1.9-slave2: 16.5G on d:, 5.50G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
  • changed frequency of l10n_nightly_scheduler from [1,9,13,17,21] to [1] in master.cfg
  • Kicked off automation:
buildbot sendchange --username=ccooper --master=localhost:9989 -m"Firefox 3.0.6build1 release" go


  • Step Tag died right away:
ASSERT: Tag::Execute(): /builds/tags/FIREFOX_3_0_5_BUILD1/cvsroot already exists? at Bootstrap/Step/ line 65.
  • tagged new bootstrap.cfg as RELEASE_AUTOMATION_M11 and sent another sendchange
  • second attempt: no problems


  • No problems

Build & Repack

  • No problems



  • Automated - no problems.

Generate Updates

  • Automated - no problems.

Publish Updates to Test Channels (betatest & releasetest)

  • Automated - no problems.

Update Verify

  • linux - no problems.
  • mac - no problems.
  • win32 - similar issues to what we saw with 3.0.5. Ben/Ted rationalized it as follows:
[12:59pm] bhearsum: the chk files are supposed to be forced updates now
[12:59pm] bhearsum: (which means the partial should contain the full file, not just a diff)
[1:00pm] ted: for precisely this reason
[1:00pm] ted: well, sort of
[1:00pm] ted: the files differ between the zip and installer, so the partial was bogus
[1:02pm] bhearsum: the MARs are generated against the zip contents ?
[1:03pm] bhearsum: well, the chk files were certainly forced
[1:03pm] ted: i think so, aren't they?
[1:03pm] ted: isn't that the base reason we hit this problem?
[1:03pm] bhearsum: i'm not entirely certain, to be honest, but it sounds right
[1:03pm] bhearsum: yeah...
[1:03pm] bhearsum: right
[1:04pm] bhearsum: so the update verify test says they're different between the new installer, and the previous installer + mar
[1:04pm] bhearsum: and the reason we force them is so teh MARs don't fail because of this
[1:04pm] bhearsum: ergo, we get these sucky errors in the update verify log, but they won't cause issues
[1:04pm] ted: yeah, so the chk files are still going to differ between the installer+partial and new installer
[1:05pm] bhearsum: even in the complete mar
[1:05pm] ted: but they should be identical in the new zip and old installer+partial
[1:05pm] bhearsum: right


  • Automated - no problems

Push updates to beta channel

# put snippets on beta
$ sudo su - cltbld
# make sure using latest version of scripts in mozilla/tools/release/bin/
$ cd bin
$ cvs update . 
$ cd /opt/aus2/snippets/staging
# note the required parameter must match what will be used with pushsnip below.
$ time ~/bin/backupsnip 20090120-Firefox-3.0.6-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20090127-1-pre-20090120-Firefox-3.0.6-beta.tar.bz2 .

real	35m49.357s
user	0m37.752s
sys	1m2.350s

$ time ~/bin/pushsnip 20090120-Firefox-3.0.6-beta
Running /usr/bin/rsync -PaO /opt/aus2/snippets/staging/20090120-Firefox-3.0.6-beta/ /opt/aus2/incoming/3

real	1m17.814s
user	0m0.140s
sys	0m5.725s

Sign Installers

Done manually using these installer-signing-instructions here.

  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0.6/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0.6/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Done.
  • Note for next release: Do not remove the Check Now bit on the Firefox-3.0.6 Products until well after the change to the rsync module (to prevent the likes of bug 464566)

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0.6/stage-merged/ /home/ftp/pub/firefox/releases/3.0.6/
  • edit the exclude file /pub/ to add the new release (3.0.6) and remove the previous release (3.0.5).

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
    • lots of 301 errors for the isohunt mirror, got pulled by sentry, wasn't grabbing the Windows builds
  • Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd /opt/aus2/snippets/staging/20090120-Firefox-3.0.6
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20090120-Firefox-3.0.6-test/$a");'

Publish Updates to Release Channel

While waiting for the formal "go," ran:

-bash-3.2$ time ~/bin/backupsnip 20090120-Firefox-3.0.6
Running /bin/tar cfvj /opt/aus2/snippets/backup/20090203-1-pre-20090120-Firefox-3.0.6.tar.bz2 .
real	46m45.374s
user	0m38.342s
sys	1m6.426s

After formal go, ran:

$ time ~/bin/pushsnip 20090120-Firefox-3.0.6
real	0m53.148s
user	0m0.108s
sys	0m4.112s


  • fix latest symlink
# cltbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.0 && ln -s 3.0.6 latest-3.0

After release, there were issues with several mirrors. nthomas had to disable all updates while he and IT investigated.

From Sam:

For those curious, the issue we had that caused us to pull updates was that a db server was mistakenly(?) taken out of rotation for bouncer, which was causing partial updates to fail. IT is investigating why this happened. Leaving updates on the release channel for users would've meant all (or maybe just most) users receiving full updates -- a problem for users (large download) and a problem for mirrors (lots more bandwidth). We'll talk more about this at the post-mortem that I'll schedule tomorrow for either later this week or early next.