Releases/Firefox 3.6.11/BuildNotes

From MozillaWiki
Jump to: navigation, search

Contents

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

jhford
Tracking bug

Signed-off Revision(s)

Build1: 620daaba9c3a

Build2: 053f07027a38

Build3: 2cfbdde77508

Tags

Build # Tag Changeset
1 GECKO19211_20100930_RELBRANCH dae5f89c4a22
FIREFOX_3_6_11_BUILD1 FIREFOX_3_6_11_RELEASE f8a11a197a82
2 GECKO19211_20100930_RELBRANCH 053f07027a38
FIREFOX_3_6_11_BUILD2 FIREFOX_3_6_11_RELEASE f8a11a197a82
3 GECKO19211_20100930_RELBRANCH 92d226250bfa
FIREFOX_3_6_11_BUILD3 FIREFOX_3_6_11_RELEASE 2cfbdde77508

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20100930133122 linux-ix-slave15 41m
Mac 20100930123656 bm-xserve09 2h02
Windows 20100930133302 mw32-ix-slave14 1h46m
2 Linux 20101001073904 linux-ix-slave15 42m
Mac 20101001074006 bm-xserve09 2h2m
Windows 20101001082906 mw32-ix-slave14 1h51m
3 Linux 20101012110043 linux-ix-slave12 1h4m
Mac 20101012104758 bm-xserve09 2h5m
Windows 20101012113537 mw32-ix-slave13 1h47m

Build 1

Done on pm:8011

  • manually clobbered linux-ix-slave17 due to using the new master and being unsure about whether the new clobberer would work
  • landed automation bumps
  • removed *.pyc in /builds/buildbot/moz2-master2
  • reconfiged master
  • kicked off automation with:
buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build1" Go

Tag

  • while tagging, I clobbered and moved:
    • linux: linux-ix-slave15, linux-ix-slave16
    • mac: bm-xserve07, bm-xserve09 (used IP due to bug 600921)
    • windows: mw32-ix-slave14, mw32-ix-slave15
  • Manually tagged buildbotcustom, buildbot-configs and tools with FIREFOX_3_6_11_RELEASE and FIREFOX_3_6_11_BUILD1

Source

No problems with source (linux-ix-slave17) or xulrunner_source (linux-ix-slave17)

Respin

There was a request to respin using a new changeset at this point

Build 2

Done on pm:8011

  • clobbered using the clobberer
  • landed 77b3526b4c3d for build 2 automation
  • removed pyc files on masters
  • reconfigured master
  • launched automation with
buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build2" Go

Tag

done on linux-ix-slave17 and 12

  • while tagging, I clobbered and moved
    • Mac: bm-xserve06, moz2-darwin9-slave20, moz2-darwin9-slave21, moz2-darwin9-slave41
    • Linux: linux-ix-slave12, mv-moz2-linux-ix-slave04,
    • Win32: mw32-ix-slave06, mw32-ix-slave11, mw32-ix-slave13
  • No problems tagging l10n repos
  • Buildbot timed out the clone of the 1.9.2 repository.
  • Attempted to kick through manually on linux-ix-slave17, but it kept hanging so I pulled it out of the pool bug 596366
  • Tried on linux-ix-slave12:
cd /builds/slave/tag
hg clone http://hg.mozilla.org/releases/mozilla-1.9.2
cd mozilla-1.9.2
hg up -C GECKO19211_20100930_RELBRANCH
hg tag -u ffxbld -r 053f07027a38 -m "Added tag FIREFOX_3_6_11_BUILD2 for changeset 053f07027a38. CLOSED TREE" FIREFOX_3_6_11_BUILD2
hg tag -u ffxbld -f -r 053f07027a38 -m "Added tag FIREFOX_3_6_11_RELEASE for changeset 053f07027a38. CLOSED TREE" FIREFOX_3_6_11_RELEASE
hg out -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.2
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.2

Succeeded on linux-ix-slave12. Started source, xulrunner_source, linux_build, macosx_build, and win32_build using "Force Build".

Source

no problems while generating firefox and xulrunner source packages

Build/Repack

  • linux build successful
  • linux l10n repacks triggered on master with
python force_release_l10n.py -m http://localhost:8011 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p linux -n jhford
  • linux repacks successful
  • win32 build successful
  • win32 l10n repacks triggered on master with
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n jhford
  • win32 failed on cs
    • retriggered by entering the following values in the force build form
      • name: jhford
      • reason: upload-failure
      • branch: cs
      • prop1: en_revision=FIREFOX_3_6_11_RELEASE
      • prop2: l10n_revision=FIREFOX_3_6_11_RELEASE
      • prop3: locale=cs
    • cs succeeded
  • all repacks other than cs succeeded first time
  • macosx build successful
  • macosx l10n repacks triggered on master with
 python force_release_l10n.py -m http://localhost:8011 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n jhford
  • all mac repacks successful
  • Verified that there were 76 locales (shipped-locales)
[ffxbld@surf ~]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/mac
[ffxbld@surf mac]$ ls | grep -v xpi | wc -l
76
[ffxbld@surf mac]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/unsigned/win32
[ffxbld@surf win32]$ ls | grep -v xpi | wc -l
76
[ffxbld@surf win32]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/linux-i686
[ffxbld@surf linux-i686]$ ls | grep -v xpi | wc -l
76

Return slaves

returned:

  • moz2-darwin9-slave20
  • moz2-darwin9-slave41
  • mw32-ix-slave06
  • mw32-ix-slave11
  • mw32-ix-slave13
  • mw32-ix-slave14
  • linux-ix-slave12
  • linux-ix-slave15
  • linux-ix-slave16

leaving:

  • mv-moz2-linux-ix-slave04
  • mw32-ix-slave14
  • bm-xserve06
  • bm-xserve07
  • bm-xserve09

XULRunner Builds

XULRunner Sign

Partner Repacks

Cancelled because we don't want them.

Unittests

These were triggered as part of the normal flow

Results

  • Linux:
    • xpcshell had a random orange bug 509970#c35
    • mochitest-other on linux-ix-slave16 failed with warnings like:
    INFO | runtests.py | Server pid: 13252
Timed out while waiting for server startup.
    • but was green on linux-ix-slave12 on rerunning
    • mochitest,reftest,crashtest were green
  • Windows:
    • mochitest-other, reftest, crashtest, xpcshell were green
  • Mac:

Sign

Removed en-US.xpi from stage, signed-build2 and unsigned-build2 after starting signing while rsync was running.

Instructions 

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.6.11
BUILD=3
TAG=FIREFOX_3_6_11_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=jhford@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r tip # repo wasn't tagged when signing was run

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

watched stdout 

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got signature verification succeeded message

L10N Verify

  • win32: all differences in metadiff happened in locales that changed in l10n-changesets
    • NOTE el did not show up in the metadiff but was changed in l10n-changesets
    • win32-el was repacked with the correct revision of this repository
hg ident -i
 in dir e:\builds\moz2_slave\win32_repack\build/releases/l10n-mozilla-1.9.2/el (timeout 1200 secs)
 watching logfiles {}
 argv: ['hg', 'ident', '-i']
 environment:
<snip>
7d52b6eaa06c
program finished with exit code 0
elapsedTime=0.234000

(builders/win32_repack/builds/166/steps/setproperty/logs/stdio)

  • mac: all differences in metadiff happened in locales that changed in l10n-changesets
    • same issue as above noticed for mac
  • linux: all differences in metadiff happened in locales that changed in l10n-changesets
    • same issue as above noticed for linux

Updates

3.6* -> 3.6.10

Generation

no problems while running the 'updates' builder

Update Verify
  • linux automation green
  • macosx automation green
  • win32 automation green

jhford -- I reconfiged your master (after things were done) -- please verify your configs before proceeding. Sorry, -aki

Push to Beta

tried 

~/bin/backupsnip 20101001-Firefox-3.6.11-beta/

but failed due to the trailing slash 

time ~/bin/backupsnip 20101001-Firefox-3.6.11-beta
real	24m12.490s
user	0m1.509s
sys	0m13.992s
~/bin/pushsnip 20101001-Firefox-3.6.11-beta ; date
<snip>
sent 3432499 bytes  received 132986 bytes  9420.04 bytes/sec
total size is 2887789  speedup is 0.81
Mon Oct  4 15:03:39 PDT 2010

3.0.19 -> 3.6.9 MU

landed update verify configs bump and Ben landed the patcher config bump for me

Generation

done on mv-moz2-linux-ix-slave04 

mkdir -p /builds/slave/3.0.19-3.6.11-major-update/snippets
cd !$
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log
# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.11 3.0.19-3.6.11
cd ../..
# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verification
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

which produced no output.

Push to AUS
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU
Enable test snippets
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101005-Firefox-3.0.19-3.6.11-MU-test
~/bin/pushsnip 20101005-Firefox-3.0.19-3.6.11-MU-test
Verify

Used mv-moz2-linux-ix-slave04, mw32-ix-slave06, bm-xserve07 

mkdir -p /builds/slave/verify/firefox-3019-3611-major/
cd /builds/slave/verify/firefox-3019-3611-major/
hg clone http://hg.mozilla.org/build/tools

cd tools/release/updates
platform=linux   # or mac
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Copied win32.log to a linux host (staging-master) to make looking at log less painful

The cleanup scripts on stage deleted all the win32 installers from the candidates directory so a patch was landed to use installers from the /releases/ directory instead of /nightly/.

Results

All differences on linux, win32 and mac were explained by the 3.0.19-3.6 update.

Respin was declared

Build 3

Landed automation bumps for build 3

set up production-master:9011 for release moved and manually clobbered linux-ix-slave12.build to pm:9011 

cd /builds/slave
rm -rf tag/ source/ updates/ linux_* xulrunner_* final_verification/ bouncer_submitter/

I was informed that the release clobberer should work in this case, so I set it but I'd also like to be sure so I did a manual clobber.

I kicked off the build with 

buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build3" Go

Tag

no issues encountered

Manually clobbered and moved bm-xserve09, mw32-ix-slave06

Mac clobber: 

rm -rf linux_l10n_verification/ macosx_* partner_repack/ win32_l10n_verification/ xulrunner_macosx_build/

Windows clobber: 

rm -rf win32_* xulrunner_win32_build

Source

no issues encountered

Build/Repack

while building, i clobbered mw32-ix-slave11, mw32-ix-slave14, linux-ix-slave14, mv-moz2-linux-ix-slave07, bm-xserve07, bm-xserve06. I then moved them onto pm:9011

linux, mac, windows builds successful.

linux, mac, windows repacks were successful

Return slaves

XULRunner Builds

no problems

XULRunner Sign

Partner Repacks

Unittests

triggered as part of the normal flow

Results

  • macosx - all green
  • linux - one random orange bug 558497, green otherwise
  • win32 - all green

Sign

Removed en-US.xpi from stage, signed-build2 and unsigned-build2 after starting signing while rsync was running.

Instructions 

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.6.11
BUILD=3
TAG=FIREFOX_3_6_11_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=jhford@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r tip # repo wasn't tagged when signing was run

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

watched stdout 

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got signature verification succeeded message

L10N Verify

Results were the same as build 2

Updates

3.6* -> 3.6.10

Generation

No problems with the automation

Update Verify

linux, mac and windows automation green

Push to Beta

On aus2-staging, ran: 

~/bin/backupsnip 20101012-Firefox-3.6.11-beta
~/bin/pushsnip 20101012-Firefox-3.6.11-beta

3.0.19 -> 3.6.9 MU

landed update verify configs and patcher configs.

Generation

done on mv-moz2-linux-ix-slave07 

mkdir -p /builds/slave/3.0.19-3.6.11-major-update/snippets
cd !$
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log
# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.11 3.0.19-3.6.11
cd ../..
# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verification
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

which produced no output

Push to AUS
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU
Enable test snippets
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101013-Firefox-3.0.19-3.6.11-MU-test
~/bin/pushsnip 20101013-Firefox-3.0.19-3.6.11-MU-test
Verify

Done on mv-moz2-linux-ix-slave07, bm-xserve06 

mkdir -p /builds/slave/verify/firefox-3019-3611-major/
cd /builds/slave/verify/firefox-3019-3611-major/
hg clone http://hg.mozilla.org/build/tools

cd tools/release/updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log
Results

linux, mac and windows differences were explained by the 3.0.19->3.6 update

Update Bouncer

Used "Force Build" to run the bouncer_submitter builder. No problems.

EU Ballot Build

# ffxbld@stage
# get tarball of unsigned EUballot builds from kev
# RelEng to sign the EUballot builds
# put copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv ../euballot-3611-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/
  • Virus scanned (separately from other builds) with:
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.11-candidates/build3/win32-EUballot

Virus Scan

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.11-candidates/build3/

Final Checks Before Push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

The first check found Solaris and OS2 builds in contrib/, which is OK. All the partner repack showed up with 644/755 permissions, which is bad (because anyone in the 'firefox' group can overwrite them). We're not using these builds for this release, but chmod'ed them correctly anyways: 

find  unsigned/partner-repacks -type d -exec chmod 755 {} \;
find  unsigned/partner-repacks -type f -exec chmod 644 {} \;

Also found:

  • OS2 builds with 666 permissions, got IT to change them to 644.
  • Solaris directories with g+s, which is OK.

Filed bug 605435 on the root issue. Re-ran the checks,

Push to mirrors

# ffxbld@stage 
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/ /pub/mozilla.org/firefox/releases/3.6.11/

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.10 with 3.6.11

EU Ballot Build Push

Pushed with: 

rsync -av /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/win32-EUballot /pub/mozilla.org/firefox/releases/3.6.11/

bug 606202 was filed to get the website updated.

Final verification (sanity check releasetest snippets)

Manually started final verification builder with "force build" on pm:8011

Push release channel snippets

As cltbld@aus2-staging: 

~/bin/backupsnip 20101012-Firefox-3.6.11
~/bin/pushsnip 20101012-Firefox-3.6.11

Re-throttle 3.0.19 release channel

bhearsum landed patch for me and pushed it live with bug 605611

Release

Update symlinks 

#ffxbld@stage
cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.11 latest-3.6

Sent the following mail to metrics@mozilla.com: 

Firefox,firefox,firefox,3.6.11,3.6
Firefox,firefox,firefox,3.6.12pre,3.6

Push 3.0.19 -> 3.6.10 MU snippets (beta and release)

~/bin/backupsnip 20101013-Firefox-3.0.19-3.6.11-MU
~/bin/pushsnip 20101013-Firefox-3.0.19-3.6.11-MU

Push XULRunner to mirrors

3.6.12/1.9.2.12 happened before this was pushed out, no need to push it.

Update XULRunner entries on wiki

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.6.11/BuildNotes&oldid=264589"