Releases/Firefox 3.6.11/BuildNotes: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 623: Line 623:


=== Final verification (sanity check releasetest snippets) ===
=== Final verification (sanity check releasetest snippets) ===
=== Push throttling changes ===


=== Release ===
=== Release ===

Revision as of 14:36, 19 October 2010

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

jhford
Tracking bug

Signed-off Revision(s)

Build1: 620daaba9c3a

Build2: 053f07027a38

Build3: 2cfbdde77508

Tags

Build # Tag Changeset
1 GECKO19211_20100930_RELBRANCH dae5f89c4a22
FIREFOX_3_6_11_BUILD1 FIREFOX_3_6_11_RELEASE f8a11a197a82
2 GECKO19211_20100930_RELBRANCH 053f07027a38
FIREFOX_3_6_11_BUILD2 FIREFOX_3_6_11_RELEASE f8a11a197a82
3 GECKO19211_20100930_RELBRANCH 92d226250bfa
FIREFOX_3_6_11_BUILD3 FIREFOX_3_6_11_RELEASE 2cfbdde77508

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20100930133122 linux-ix-slave15 41m
Mac 20100930123656 bm-xserve09 2h02
Windows 20100930133302 mw32-ix-slave14 1h46m
2 Linux 20101001073904 linux-ix-slave15 42m
Mac 20101001074006 bm-xserve09 2h2m
Windows 20101001082906 mw32-ix-slave14 1h51m
3 Linux 20101012110043 linux-ix-slave12 1h4m
Mac 20101012104758 bm-xserve09 2h5m
Windows 20101012113537 mw32-ix-slave13 1h47m

Build 1

Done on pm:8011

  • manually clobbered linux-ix-slave17 due to using the new master and being unsure about whether the new clobberer would work
  • landed automation bumps
  • removed *.pyc in /builds/buildbot/moz2-master2
  • reconfiged master
  • kicked off automation with:
buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build1" Go

Tag

  • while tagging, I clobbered and moved:
    • linux: linux-ix-slave15, linux-ix-slave16
    • mac: bm-xserve07, bm-xserve09 (used IP due to bug 600921)
    • windows: mw32-ix-slave14, mw32-ix-slave15
  • Manually tagged buildbotcustom, buildbot-configs and tools with FIREFOX_3_6_11_RELEASE and FIREFOX_3_6_11_BUILD1

Source

No problems with source (linux-ix-slave17) or xulrunner_source (linux-ix-slave17)

Respin

There was a request to respin using a new changeset at this point

Build 2

Done on pm:8011

  • clobbered using the clobberer
  • landed 77b3526b4c3d for build 2 automation
  • removed pyc files on masters
  • reconfigured master
  • launched automation with
buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build2" Go

Tag

done on linux-ix-slave17 and 12

  • while tagging, I clobbered and moved
    • Mac: bm-xserve06, moz2-darwin9-slave20, moz2-darwin9-slave21, moz2-darwin9-slave41
    • Linux: linux-ix-slave12, mv-moz2-linux-ix-slave04,
    • Win32: mw32-ix-slave06, mw32-ix-slave11, mw32-ix-slave13
  • No problems tagging l10n repos
  • Buildbot timed out the clone of the 1.9.2 repository.
  • Attempted to kick through manually on linux-ix-slave17, but it kept hanging so I pulled it out of the pool bug 596366
  • Tried on linux-ix-slave12:
cd /builds/slave/tag
hg clone http://hg.mozilla.org/releases/mozilla-1.9.2
cd mozilla-1.9.2
hg up -C GECKO19211_20100930_RELBRANCH
hg tag -u ffxbld -r 053f07027a38 -m "Added tag FIREFOX_3_6_11_BUILD2 for changeset 053f07027a38. CLOSED TREE" FIREFOX_3_6_11_BUILD2
hg tag -u ffxbld -f -r 053f07027a38 -m "Added tag FIREFOX_3_6_11_RELEASE for changeset 053f07027a38. CLOSED TREE" FIREFOX_3_6_11_RELEASE
hg out -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.2
hg push -e 'ssh -l ffxbld -i ~cltbld/.ssh/ffxbld_dsa' -f ssh://hg.mozilla.org/releases/mozilla-1.9.2

Succeeded on linux-ix-slave12. Started source, xulrunner_source, linux_build, macosx_build, and win32_build using "Force Build".

Source

no problems while generating firefox and xulrunner source packages

Build/Repack

  • linux build successful
  • linux l10n repacks triggered on master with
python force_release_l10n.py -m http://localhost:8011 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p linux -n jhford
  • linux repacks successful
  • win32 build successful
  • win32 l10n repacks triggered on master with
python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p win32 -n jhford
  • win32 failed on cs
    • retriggered by entering the following values in the force build form
      • name: jhford
      • reason: upload-failure
      • branch: cs
      • prop1: en_revision=FIREFOX_3_6_11_RELEASE
      • prop2: l10n_revision=FIREFOX_3_6_11_RELEASE
      • prop3: locale=cs
    • cs succeeded
  • all repacks other than cs succeeded first time
  • macosx build successful
  • macosx l10n repacks triggered on master with
 python force_release_l10n.py -m http://localhost:8011 -t FIREFOX_3_6_11_RELEASE -v -b releases/mozilla-1.9.2 -p macosx -n jhford
  • all mac repacks successful
  • Verified that there were 76 locales (shipped-locales)
[ffxbld@surf ~]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/mac
[ffxbld@surf mac]$ ls | grep -v xpi | wc -l
76
[ffxbld@surf mac]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/unsigned/win32
[ffxbld@surf win32]$ ls | grep -v xpi | wc -l
76
[ffxbld@surf win32]$ cd /home/ftp/pub/firefox/nightly/3.6.11-candidates//build2/linux-i686
[ffxbld@surf linux-i686]$ ls | grep -v xpi | wc -l
76

Return slaves

returned:

  • moz2-darwin9-slave20
  • moz2-darwin9-slave41
  • mw32-ix-slave06
  • mw32-ix-slave11
  • mw32-ix-slave13
  • mw32-ix-slave14
  • linux-ix-slave12
  • linux-ix-slave15
  • linux-ix-slave16

leaving:

  • mv-moz2-linux-ix-slave04
  • mw32-ix-slave14
  • bm-xserve06
  • bm-xserve07
  • bm-xserve09

XULRunner Builds

XULRunner Sign

Partner Repacks

Cancelled because we don't want them.

Unittests

These were triggered as part of the normal flow

Results

  • Linux:
    • xpcshell had a random orange bug 509970#c35
    • mochitest-other on linux-ix-slave16 failed with warnings like:
    INFO | runtests.py | Server pid: 13252
Timed out while waiting for server startup.
    • but was green on linux-ix-slave12 on rerunning
    • mochitest,reftest,crashtest were green
  • Windows:
    • mochitest-other, reftest, crashtest, xpcshell were green
  • Mac:

Sign

Removed en-US.xpi from stage, signed-build2 and unsigned-build2 after starting signing while rsync was running.

Instructions 

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.6.11
BUILD=3
TAG=FIREFOX_3_6_11_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=jhford@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r tip # repo wasn't tagged when signing was run

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

watched stdout 

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got signature verification succeeded message

L10N Verify

  • win32: all differences in metadiff happened in locales that changed in l10n-changesets
    • NOTE el did not show up in the metadiff but was changed in l10n-changesets
    • win32-el was repacked with the correct revision of this repository
hg ident -i
 in dir e:\builds\moz2_slave\win32_repack\build/releases/l10n-mozilla-1.9.2/el (timeout 1200 secs)
 watching logfiles {}
 argv: ['hg', 'ident', '-i']
 environment:
<snip>
7d52b6eaa06c
program finished with exit code 0
elapsedTime=0.234000

(builders/win32_repack/builds/166/steps/setproperty/logs/stdio)

  • mac: all differences in metadiff happened in locales that changed in l10n-changesets
    • same issue as above noticed for mac
  • linux: all differences in metadiff happened in locales that changed in l10n-changesets
    • same issue as above noticed for linux

Updates

3.6* -> 3.6.10

Generation

no problems while running the 'updates' builder

Update Verify
  • linux automation green
  • macosx automation green
  • win32 automation green

jhford -- I reconfiged your master (after things were done) -- please verify your configs before proceeding. Sorry, -aki

Push to Beta

tried 

~/bin/backupsnip 20101001-Firefox-3.6.11-beta/

but failed due to the trailing slash 

time ~/bin/backupsnip 20101001-Firefox-3.6.11-beta
real	24m12.490s
user	0m1.509s
sys	0m13.992s
~/bin/pushsnip 20101001-Firefox-3.6.11-beta ; date
<snip>
sent 3432499 bytes  received 132986 bytes  9420.04 bytes/sec
total size is 2887789  speedup is 0.81
Mon Oct  4 15:03:39 PDT 2010

3.0.19 -> 3.6.9 MU

landed update verify configs bump and Ben landed the patcher config bump for me

Generation

done on mv-moz2-linux-ix-slave04 

mkdir -p /builds/slave/3.0.19-3.6.11-major-update/snippets
cd !$
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log
# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.11 3.0.19-3.6.11
cd ../..
# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verification
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

which produced no output.

Push to AUS
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101005-Firefox-3.0.19-3.6.11-MU
Enable test snippets
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101005-Firefox-3.0.19-3.6.11-MU-test
~/bin/pushsnip 20101005-Firefox-3.0.19-3.6.11-MU-test
Verify

Used mv-moz2-linux-ix-slave04, mw32-ix-slave06, bm-xserve07 

mkdir -p /builds/slave/verify/firefox-3019-3611-major/
cd /builds/slave/verify/firefox-3019-3611-major/
hg clone http://hg.mozilla.org/build/tools

cd tools/release/updates
platform=linux   # or mac
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Copied win32.log to a linux host (staging-master) to make looking at log less painful

The cleanup scripts on stage deleted all the win32 installers from the candidates directory so a patch was landed to use installers from the /releases/ directory instead of /nightly/.

Results

All differences on linux, win32 and mac were explained by the 3.0.19-3.6 update.

Respin was declared

Build 3

Landed automation bumps for build 3

set up production-master:9011 for release moved and manually clobbered linux-ix-slave12.build to pm:9011 

cd /builds/slave
rm -rf tag/ source/ updates/ linux_* xulrunner_* final_verification/ bouncer_submitter/

I was informed that the release clobberer should work in this case, so I set it but I'd also like to be sure so I did a manual clobber.

I kicked off the build with 

buildbot sendchange --username=jhford --master=localhost:9011 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.11 build3" Go

Tag

no issues encountered

Manually clobbered and moved bm-xserve09, mw32-ix-slave06

Mac clobber: 

rm -rf linux_l10n_verification/ macosx_* partner_repack/ win32_l10n_verification/ xulrunner_macosx_build/

Windows clobber: 

rm -rf win32_* xulrunner_win32_build

Source

no issues encountered

Build/Repack

while building, i clobbered mw32-ix-slave11, mw32-ix-slave14, linux-ix-slave14, mv-moz2-linux-ix-slave07, bm-xserve07, bm-xserve06. I then moved them onto pm:9011

linux, mac, windows builds successful.

linux, mac, windows repacks were successful

Return slaves

XULRunner Builds

XULRunner Sign

Partner Repacks

Unittests

triggered as part of the normal flow

Results

  • macosx - all green
  • linux - one random orange bug 558497, green otherwise
  • win32 - all green

Sign

Removed en-US.xpi from stage, signed-build2 and unsigned-build2 after starting signing while rsync was running.

Instructions 

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.6.11
BUILD=3
TAG=FIREFOX_3_6_11_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=jhford@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r tip # repo wasn't tagged when signing was run

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

watched stdout 

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got signature verification succeeded message

L10N Verify

Results were the same as build 2

Updates

3.6* -> 3.6.10

Generation

No problems with the automation

Update Verify

linux, mac and windows automation green

Push to Beta

On aus2-staging, ran: 

~/bin/backupsnip 20101012-Firefox-3.6.11-beta
~/bin/pushsnip 20101012-Firefox-3.6.11-beta

3.0.19 -> 3.6.9 MU

landed update verify configs and patcher configs.

Generation

done on mv-moz2-linux-ix-slave07 

mkdir -p /builds/slave/3.0.19-3.6.11-major-update/snippets
cd !$
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log
# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.11 3.0.19-3.6.11
cd ../..
# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verification
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

which produced no output

Push to AUS
cd /builds/slave/3.0.19-3.6.11-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.11
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20101013-Firefox-3.0.19-3.6.11-MU
Enable test snippets
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20101013-Firefox-3.0.19-3.6.11-MU-test
~/bin/pushsnip 20101013-Firefox-3.0.19-3.6.11-MU-test
Verify

Done on mv-moz2-linux-ix-slave07, bm-xserve06 

mkdir -p /builds/slave/verify/firefox-3019-3611-major/
cd /builds/slave/verify/firefox-3019-3611-major/
hg clone http://hg.mozilla.org/build/tools

cd tools/release/updates
platform=linux   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log
Results

linux, mac and windows differences were explained by the 3.0.19->3.6 update

Update Bouncer

Used "Force Build" to run the bouncer_submitter builder. No problems.

EU Ballot Build

Virus Scan

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.11-candidates/build3/

Final Checks Before Push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/
find . ! -user ffxbld
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

The first check found Solaris and OS2 builds in contrib/, which is OK. All the partner repack showed up with 644/755 permissions, which is bad (because anyone in the 'firefox' group can overwrite them). We're not using these builds for this release, but chmod'ed them correctly anyways: 

find  unsigned/partner-repacks -type d -exec chmod 755 {} \;
find  unsigned/partner-repacks -type f -exec chmod 644 {} \;

Also found:

  • OS2 builds with 666 permissions, got IT to change them to 644.
  • Solaris directories with g+s, which is OK.

Filed bug 605435 on the root issue. Re-ran the checks,

Push to mirrors

# ffxbld@stage 
rsync -av --exclude="*reporter*" --exclude=*.log --exclude=*.txt --exclude=*unsigned* /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/ /pub/mozilla.org/firefox/releases/3.6.11/

# cltbld@stage
vim -n /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.10 with 3.6.11

EU Ballot Build Push

Done separately from the original push because they weren't ready when we pushed to mirrors. Staged with: 

# ffxbld@stage
# get copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv ../euballot-3611-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/

Virus Scanned with: 

clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.6.11-candidates/build3/win32-EUballot

Pushed with: 

rsync -av /pub/mozilla.org/firefox/nightly/3.6.11-candidates/build3/win32-EUballot /pub/mozilla.org/firefox/releases/3.6.11/

Final verification (sanity check releasetest snippets)

Release

Push EU Ballot build

Push 3.0.19 -> 3.6.10 MU snippets (beta and release)

Push XULRunner to mirrors

Update XULRunner entries on wiki

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.6.11/BuildNotes&oldid=261734"