Releases/Firefox 3.6.6/BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers


Tracking bug

Signed-off Revision(s)



On releases/mozilla-1.9.2:

Build # Tag Changeset
1 GECKO1924_20100413_RELBRANCH 2e4643219bd6
FIREFOX_3_6_6_BUILD1 FIREFOX_3_6_6_RELEASE c07c6c1b5071

Build data

Build # Type Build ID Build machine
1 Linux 20100625222733 mv-moz2-linux-ix-slave06
Mac 20100625223402 bm-xserve18
Windows 20100625231939 mw32-ix-slave06


Build 1

  • Clobbered 'Any release' on pm with clobberer.
  • Started pm master (had been shutdown because it's unused).
  • Moved the following slaves to pm:
    • mw32-ix-slave03,04,05,06
    • bm-xserve17,18,19,21,22
    • mv-moz2-linux-ix-slave02,03,05,06,07
  • Landed automation config
  • Reconfiged pm
  • Closed the Firefox3.6 tree
  • Started the automation:
buildbot sendchange --username=catlee --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.6 build1" Go


No problems. Very fast! 12 minutes to complete!

Opened Firefox3.6 tree again.


No problems

XULRunner Source

No problems


  • linux: no problems
  • macosx: no problems
  • win32
    • cs repack failed while doing "make l10n-upload-cs":
d:/mozilla-build/python25/python2.5.exe ../../build/ --base-path ../../dist "../../dist/win32/cs/" ../../dist/win32/xpi/cs.xpi
Uploading e:\builds\moz2_slave\win32_repack\build\mozilla-1.9.2\dist\win32\cs\
Uploading e:\builds\moz2_slave\win32_repack\build\mozilla-1.9.2\dist\win32\xpi\cs.xpi
Running post-upload command: -p firefox -v 3.6.6 -n 1 --release-to-candidates-dir
Upload complete
d:/mozilla-build/python25/python2.5.exe ../../build/ --base-path ../../dist ../../dist/update/win32/cs/firefox-3.6.6.complete.mar
Uploading e:\builds\moz2_slave\win32_repack\build\mozilla-1.9.2\dist\update\win32\cs\firefox-3.6.6.complete.mar
Running post-upload command: -p firefox -v 3.6.6 -n 1 --release-to-candidates-dir
Upload complete
d:/mozilla-build/python25/python2.5.exe ../../build/ --base-path ../../dist "../../dist/win32/cs/Firefox Setup 3.6.6.exe"
Read from remote host Connection reset by peer
lost connection
Uploading e:\builds\moz2_slave\win32_repack\build\mozilla-1.9.2\dist\win32\cs\Firefox Setup 3.6.6.exe
Encountered error while uploading
Command '['scp', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', '/e/builds/moz2_slave/win32_repack/build/mozilla-1.9.2/dist/win32/cs/Firefox Setup 3.6.6.exe', '']' returned non-zero exit status 1
make: *** [l10n-upload-cs] Error 2

Rebuilt using this instructions:

[cltbld@production-master buildbot-helpers]$ python -m http://localhost:8010 -t FIREFOX_3_6_6_RELEASE -v -s locales-file -n
Forcing http://localhost:8010/builders/win32_repack/force with params: {'username': '', 'prop3value': 'FIREFOX_3_6_6_RELEASE', 'prop2value': 'cs', 'comments': 'Unknown', 'prop2name': 'locale', 'prop1value': 'FIREFOX_3_6_6_RELEASE', 'prop3name': 'l10n_revision', 'branch': 'cs', 'prop1name': 'en_revision'}
[cltbld@production-master buildbot-helpers]$ cat locales-file 
cs win32

Partner Repacks

Can't do these from the automation until bug 554321 is resolved.

Unit tests

  • macosx: green
  • win32: green
  • linux: orange linux_test xpcshell (bug 507239?)
TEST-UNEXPECTED-FAIL | /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/tests/test_places/bookmarks/test_395593.js | test failed (with xpcshell return code: 0), see following log:
  TEST-INFO | (xpcshell/head.js) | test 1 pending
TEST-UNEXPECTED-FAIL | /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/tests/test_places/bookmarks/test_395593.js | 1 == 3 - See following stack:
JS frame :: /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/head.js :: do_throw :: line 200
JS frame :: /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/head.js :: do_check_eq :: line 230
JS frame :: /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/tests/test_places/bookmarks/test_395593.js :: run_test :: line 84
JS frame :: /builds/slave/release-linux-unittest-xpcshell/build/xpcshell/head.js :: _execute_test :: line 128
JS frame :: -e :: <TOP_LEVEL> :: line 1
TEST-INFO | (xpcshell/head.js) | exiting test



  • Removed en-US.xpi from stage
set -x
set -e


mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}

cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .


  • All targets ran fine.
  • logs look fine
  • Ran postsign:

Generate updates and push betatest snippets

Failed when generating a partial update for the fa locale on mac.

Extracting /builds/slave/updates/build/temp/firefox/3.6.4/ftp/firefox-3.6.4.fa.mac.complete.mar to /builds/slave/updates/build/tmpOUjAmC-fastmode/97/from
Extracting /builds/slave/updates/build/temp/firefox/3.6.6/ftp/firefox-3.6.6.fa.mac.complete.mar to /builds/slave/updates/build/tmpOUjAmC-fastmode/97/to
skipping diff: Contents/Info.plist
diffing: Contents/MacOS/XUL
bzip2: Data integrity error when decompressing.
	Input file = /builds/slave/updates/build/tmpOUjAmC-fastmode/97/to/Contents/MacOS/XUL.bz2, output file = /builds/slave/updates/build/tmpOUjAmC-fastmode/97/to/Contents/MacOS/XUL
It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

This becomes clearer if you check:

# on stage
$ pwd 
$ ls -l mac/fa/firefox-3.6.6.complete.mar 
-rw-r--r-- 1 ffxbld firefox 18840313 Jun 26 00:48 mac/fa/firefox-3.6.6.complete.mar
$ sha1sum mac/fa/firefox-3.6.6.complete.mar 
e696cd6683e3f164f9a0b13f6cbb105580b8d16f  mac/fa/firefox-3.6.6.complete.mar

# on the slave (mv-moz2-linux-ix-slave07)
$ pwd
$ ls -l firefox-3.6.6.fa.mac.complete.mar 
-rw-rw-r-- 1 cltbld cltbld 18840313 Jun 26 00:48 firefox-3.6.6.fa.mac.complete.mar
$ sha1sum firefox-3.6.6.fa.mac.complete.mar 
b52f66027f941e5028f0692e9ff64626525d51ca  firefox-3.6.6.fa.mac.complete.mar
$ wget -q -O- | sha1sum
b52f66027f941e5028f0692e9ff64626525d51ca  -

So the file size is correct but the hash is not, and the Castro cache strikes again. Fix:

  • move /builds/slave/updates to updates.broken (instead of clobber to leave for debugging)
  • remove file from proxy
# bc-proxy01
/usr/sbin/squidclient -m purge
  • force build on 'updates' (and expect it to fail the diff_patcher_config step)

L10N Verification

No problems

Update verify

Forced due to failed updates builder

  • Linux: PASS
  • Windows: PASS
  • Mac: PASS

3.5.10 -> 3.6.6 MU Generation


Update pm02, reconfig

Hit Force Build on major update builder

No problems on major_update, *_major_update_verify got triggered without problems.

  • All platforms
    • no update for mn since that doesn't exist for 3.6.4
    • sl leaves these searchplugins behind (they were removed between 3.6 and 3.6.2)
  • Linux: PASS, no other differences
  • Mac: PASS; had inconsequential channel-prefs.js differences, and plugin directories:
   Only in source/ Plug-Ins
   Contents of source/ dir only in source or target
   158131022        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
   158131023        0 drwxr-xr-x    4 cltbld   staff         136 Jun 26 07:34 source/
   158131027        0 drwxr-xr-x    2 cltbld   staff          68 Jun 26 07:34 source/
   158131028        0 drwxr-xr-x    3 cltbld   staff         102 May  4 10:31 source/
   158131030        0 drwxr-xr-x    3 cltbld   staff         102 Jun 26 07:34 source/
   158131034        0 drwxr-xr-x    2 cltbld   staff          68 Jun 26 07:34 source/
  • Win32 - PASS, with the usual issues differences for .autoreg, freebl3.chk, nssdbm3.chk, removed-files, softokn3.chk

3.0.19 -> 3.6.6 MU

Snippet Generation

on mv-moz2-linux-ix-slave02:

mkdir -p /builds/slave/3.0.19-3.6.6-major-update/snippets
cd /builds/slave/3.0.19-3.6.6-major-update/snippets

cvs co -d patcher-configs mozilla/tools/patcher-configs
cvs co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

# build tools
./ --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./ --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.6 3.0.19-3.6.6
cd ../..

# Create partial patches and snippets
./ --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verify

Check that releasetest = beta = release

cd temp/firefox/3.0.19-3.6.6

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'
Push to AUS
cd /builds/3.0.19-3.6.6-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.6
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/
Enable test snippets
~/bin/backupsnip 20100626-Firefox-3.0.19-3.6.6-MU-test
~/bin/pushsnip   20100626-Firefox-3.0.19-3.6.6-MU-test
Update Verify

Ran update verify on mv-moz2-linux-ix-slave02, bm-xserve18, mw32-ix-slave03 as follows:

 mkdir -p /builds/slave/verify/firefox-3019-366-major/
 cd /builds/slave/verify/firefox-3019-366-major/
 hg clone
 cd tools/release/updates
 platform=linux   # or mac or win32
 ./ -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Checked the logs carefully and found the same differences documented for 3.0.19 -> 3.6's major update.

Socorro product data updates

Went to, clicked on 'add new product version', and entered Firefox 3.6.6, on the 1.9.2 branch, and the default dates.

Update Bouncer

  • Added manually.

Final checks before push

# ffxbld@stage
cd /pub/
find . ! -user ffxbld #nothing showed up, as expected.
find . ! -group firefox #nothing showed up, as expected.
find . -type f ! -perm 644 # unsigned/partner-repacks have 664 permission
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' # unsigned/partner-repacks have 664 permission
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*' # nothing showed up, as expected.

Virus Scan

[ffxbld@surf build1]$ cd /pub/
[ffxbld@surf build1]$ clamdscan -m /mnt/netapp/stage/
/mnt/netapp/stage/ OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 3632.544 sec (60 m 32 s)

EU Ballot Build

This is a win32 partner build for the EU browser ballot.

After the builds are created (this time by Kev), and passed QA, they were signed and moved into the candidates dir

# ffxbld@stage
# get copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv ../euballot-366-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/

The builds will be pushed when the other ones do, in 'push to mirrors'.

Scanned these files since they were pushed after the previous scan was done:

clamdscan -m /mnt/netapp/stage/

/mnt/netapp/stage/ OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 36.397 sec (0 m 36 s)

Push to mirrors

# ffxbld@stage
rsync -av --exclude=beta --exclude=*.log --exclude=*.txt --exclude=*unsigned*   --exclude=* --exclude=*.tests.tar.bz2*   --exclude=*partner-repacks*   /pub/   /pub/
sent 6765077458 bytes  received 24977 bytes  7124910.41 bytes/sec
total size is 6764159455  speedup is 1.00
  • Update cltbld@stage:/pub/ to include 3.6.6 instead of 3.6.4.

Final Verification

Run the 'final_verification' builder on pm.b.m.o

  • click "force build" from buildbot UI. Do not set any properties, just fill in name and click "force build".
    • First time had problems with Reran. 2nd run ran green.

Push updates to release channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100625-Firefox-3.6.6
real    11m12.254s
user    0m0.916s
sys     0m9.166s
~/bin/backupsnip 20100626-Firefox-3.5.10-3.6.6-MU
real    0m32.204s
user    0m0.055s
sys     0m0.608s
~/bin/backupsnip 20100626-Firefox-3.0.19-3.6.6-MU
real    0m18.663s
user    0m0.059s
sys     0m0.498s

3.6.x -> 3.6.6 minor updates:

~/bin/pushsnip 20100625-Firefox-3.6.6
sent 1249137 bytes  received 59828 bytes  17337.28 bytes/sec
total size is 1026485  speedup is 0.78

3.5.10 major update (unadvertised):

~/bin/pushsnip   20100626-Firefox-3.5.10-3.6.6-MU
sent 218861 bytes  received 9788 bytes  26899.88 bytes/sec
total size is 180234  speedup is 0.79

3.0.19 major update (unadvertised):

~/bin/pushsnip   20100626-Firefox-3.0.19-3.6.6-MU
sent 180559 bytes  received 8028 bytes  17960.67 bytes/sec
total size is 148838  speedup is 0.79


# ffxbld@stage
cd /pub/
rm latest-3.6
ln -s 3.6.6 latest-3.6

Push EU Ballot Build

bug 575001 for IT to update redirect on download.m.o and update

Push updates to beta channel

~/bin/backupsnip 20100625-Firefox-3.6.6-beta
real	14m0.911s
user	0m1.145s
sys	0m11.482s
~/bin/pushsnip 20100625-Firefox-3.6.6-beta
sent 2081309 bytes  received 90188 bytes  16144.96 bytes/sec
total size is 1750363  speedup is 0.81
 ~/bin/backupsnip   20100626-Firefox-3.5.10-3.6.6-MU-beta
real    0m41.282s
user    0m0.040s
sys     0m0.505s
 ~/bin/backupsnip   20100626-Firefox-3.0.19-3.6.6-MU-beta
real    0m41.282s
user    0m0.040s
sys     0m0.505s
~/bin/pushsnip   20100626-Firefox-3.5.10-3.6.6-MU-beta
sent 219527 bytes  received 11120 bytes  21966.38 bytes/sec
total size is 180234  speedup is 0.78
~/bin/pushsnip   20100626-Firefox-3.0.19-3.6.6-MU-beta
sent 180013 bytes  received 8028 bytes  19793.79 bytes/sec
total size is 148838  speedup is 0.79

XULRunner build

No problems.

  • Copied xrbld_dsa key from a production slave (chmod 600)

Used the following script:

set -x
set -e


mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}
cd -

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=* \${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
rsync -av unsigned-build${BUILD}/unsigned/ \

make checksum-files create-sigs stage verify-sigs \

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
Push XulRunner to mirrors
# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=* \
  /pub/ \

Edited cltbld@stage:/pub/ and added

- xulrunner/releases/1.9.2
Update XulRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!


Moved slaves back to their masters (commented buildmaster_host lines in buildbot.tac were very helpful), and shutdown master on pm.