Releases/Firefox 2.0.0.16-3.0.1 MU:BuildNotes

Release Engineers

Nick Thomas (major update test)
Ben Hearsum (real snippet generation, null partner updates)

Bugs

bug 442105 - tracking bug
bug 450863 - null partner updates

Generate Major Update Snippets

# ssh cltbld@prometheus-vm.build.mozilla.org
cd /builds/2.0.0.16-major-update/snippets
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d MozBuild mozilla/tools/release/MozBuild
cd ../
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs

cd patcher
export CVSROOT=:ext:ffxbld@cvs.mozilla.org:/cvsroot

# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R4 --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.0.1 2.0.0.16-3.0.1

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Generate Null Partner Snippets

Create Partner Channel List

Starting with https://bugzilla.mozilla.org/attachment.cgi?id=334332

# Removed any channels that don't start with 'release-' from the list
# Now, we only have a list of the real release channels for partner builds
# we need to add beta and test channels to the list:
for i in `cat partner-update-channels`; do basechan=`echo $i | sed -e 's/release-//'` && echo "beta-$basechan" >> partner-update-channels && echo "betatest-$basechan" >> partner-update-channels && echo "releasetest-$basechan" >> partner-update-channels; done
# now, we have all the channels we need except for the 4 listed at the start
# because the real release channels aren't prepended with 'release' we will do this different:
for i in [list of channels that don't start with release-]; do echo $i >> partner-update-channels && echo "beta-$i" >> partner-update-channels && echo "betatest-$i" >> partner-update-channels && echo "releasetest-$i" >> partner-update-channels; done
# now we have a complete list of partner channels to generate snippets for

Generate null snippets

updated null_partners.sh with 2.0.0.16 buildids, and enabled it to read a list of partner channels from an external file - rev 1.3

# ssh cltbld@prometheus-vm.build.mozilla.org
cd /builds/2.0.0.16-major-update/snippets/patcher/temp/firefox/2.0.0.16-3.0.1
bash ../../../one-offs/null_partners.sh ../../../partner-update-channels

Create beta snippet dir

rsync -a aus2/ aus2.beta/

Trim snippets from release, beta, test dirs

Our 'aus2.test' directory should only contain 'releasetest-*' and 'betatest-*' snippets. Similarly, 'aus2.beta' should only contain 'beta-*' snippets and 'aus2' should only have 'release-*'

# beta
find aus2.beta/ -type d -name "release*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "releasetest*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "betatest*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
# these ones get rid of those funny channels
find aus2.beta/ -type d -name "default*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "nightly*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "Windows*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
# release 
find aus2/ -type d -name "beta*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2/ -type d -name "releasetest*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2/ -type d -name "betatest*" -exce rm -rfv {} \; 2>&1 | tee -a release_remove.log
# test
find aus2.test/ -type d -name "release-*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "beta-*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
# these ones get rid of those funny channels
find aus2.test/ -type d -name "default*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "nightly*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "Windows*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log

Move null snippets to their own directory

mkdir aus2.null
rsync -av aus2/ aus2.null/
rsync -av aus2.beta/ aus2.null/
rsync -av aus2.test/ aus2.null/
# trim non-partner snippets from the null directory
find aus2.null -type d -mindepth 6 -maxdepth 6 | egrep '.*\/(release|releasetest|beta|betatest)$' | xargs rm -rfv 2>&1 | tee -a ../null_remove.log
# trim partner snippets from the other directories
find aus2.test -type d -mindepth 6 -maxdepth 6 | egrep '.*\/(release|beta)test-.*$' | xargs rm -rfv 2>&1 | tee -a ../test_remove.log
find aus2.beta -type d -mindepth 6 -maxdepth 6 | egrep '.*\/beta-.*$' | xargs rm -rfv 2>&1 | tee -a ../beta_remove.log
find aus -type d -mindepth 6 -maxdepth 6 | egrep -v '.*\/release$' | xargs rm -rfv 2>&1 | tee -a ../release_remove.log 

Verify correct snippets are in correct directories

Make sure that only betatest/releasetest snippets are in aus2.test, beta snippets in aus2.beta, and release snippets in aus2
# aus2.null
find aus2.null -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep '^(release-*|releasetest-*|beta-*|betatest-*)'
# aus2.test
find aus2.test -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^(beta|release)test$'
# no output
# beta
find aus2.beta -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^beta$'
# no output
# release
find aus2 -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^release$'

Push snippet directories to aus2-staging

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-beta cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-beta
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-test cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-null cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-null

Push null & test snippets live

cd /opt/aus2/snippets/staging
~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-null
~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-null
~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-test
~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-test

Fix up null partner snippets to target 2.0.0.14

After partner snippets were pushed I was informed that the latest partner builds we shipped were for 2.0.0.14. Therefore, the snippets pushed are useless. To fix them up, did the following:

# cltbld@prometheus-vm
cd /builds/2.0.0.16-major-update/snippets/patcher/temp/firefox/2.0.0.16-3.0.1
cd aus2.null/Firefox
mv 2.0.0.16 2.0.0.14
cd 2.0.0.14/Darwin_Universal-gcc3
mv 2008070205 2008040413
cd ../Linux_x86-gcc3
mv 2008070204 2008040414
cd ../WINNT_x86-msvc
mv 2008070205 2008040413

Also had to generate snippets for another partner channel for 2.0.0.16 (https://bugzilla.mozilla.org/show_bug.cgi?id=450863#c5) To do that, did the following:

cd /builds/2.0.0.16-major-update/snippets/patcher
echo "release-cck-asus-google" >> additional-partner-channels
echo "releasetest-cck-asus-google" >> additional-partner-channels
echo "beta-cck-asus-google" >> additional-partner-channels
echo "betatest-cck-asus-google" >> aditional-partner-channels
cd temp/firefox/2.0.0.16-3.0.1
# modified null_partners.sh to only generage snippets in aus2.null
bash ../../../one-offs/null_partners.sh ../../../additional-partner-channels

Did a spot check of aus2.null, looked good. Pushed new snippets to aus2-staging:

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.null cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-null-partner-snippets

Verify null partner updates

Modified create_aus_urls.txt to use a different set of locales for each platform and read partner channels from a file. Lists of locales for each platform are in ${platform}_locales.txt. List of partner channels in partner-update-channels. Generated URLs as follows:

./create_aus_urls.sh

Used test_null_updates.sh to test all snippets:

./test_null_updates.sh mu-2.0.0.14-3.0.1-null-urls.txt 2>&1 | tee before.log
# grep the log for any non-passes
grep -v '^PASS'
# no output - everything is good!

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-20016-301-real-major/
cd /builds/verify/firefox-20016-301-real-major/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d updates mozilla/testing/release/updates/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/

# get patch to trawl "Only in" directories
cd common
# on mac & linux
curl -s https://bugzilla.mozilla.org/attachment.cgi?id=304252 | patch -p0
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=304252
patch -p0 < patch
 cd ../updates

./verify.sh -c moz18-firefox-$platform-major.cfg 2>&1 | tee $platform.log
# had to kick along some of them because they hit slow bouncer mirrors.
# did this by killing the wget process (which makes it retry)

Add 'sl' MU snippets + null partner snippets

sl was missed in the patcher config file for the original run :(. To fix, re-ran steps 3 and 4.2 from above. Then separated out beta snippets into their own directory:

rsync aus2/ aus2.beta/
find aus2/ -type d -name "beta*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2.beta/ -type d -name "release*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log

Then synced sl snippets over to aus2-staging:

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-test/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-beta/

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.null/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-null-partner-snippets/

Pushed null and test snippets:

~/bin/pushsnip 20080821-Firefox-MU-sl-only-null-partner-snippets
~/bin/pushsnip 20080821-Firefox-MU-sl-only-test

Separate ja-JP-mac snippets into a separate directory

# cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging
rsync -av 20080820-Firefox-2.0.0.16-3.0.1-MU 20080821-Firefox-MU-ja-JP-mac-only
rsync -av 20080820-Firefox-2.0.0.16-3.0.1-MU-beta 20080821-Firefox-MU-ja-JP-mac-only-beta
cd 20080821-Firefox-MU-ja-JP-mac-only
mv 20080820-Firefox-2.0.0.16-3.0.1-MU/Firefox .
rmdir 20080820-Firefox-2.0.0.16-3.0.1-MU
find . -mindepth 5 -maxdepth 5 | grep -v ja-JP-mac | xargs rm -rfv
cd 20080821-Firefox-MU-ja-JP-mac-only-beta
mv 20080820-Firefox-2.0.0.16-3.0.1-MU/Firefox-beta .
rmdir 20080820-Firefox-2.0.0.16-3.0.1-MU-beta
find . -mindepth 5 -maxdepth 5 | grep -v ja-JP-mac | xargs rm -rfv
# now remove ja-JP-mac from the original directories
cd ../20080820-Firefox-2.0.0.16-3.0.1-MU/
find . -maxdepth 5 -mindepth 5 | grep 'ja-JP-mac' | xargs rm -rfv
cd ../20080820-Firefox-2.0.0.16-3.0.1-MU-beta/
find . -maxdepth 5 -mindepth 5 | grep 'ja-JP-mac' | xargs rm -rfv

HOW TO PUSH TO BETA

To push to beta the following directories need to be pushsnip'ed on aus2-staging:

# contains every locale except 'sl'
/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-beta
# only contains 'sl'
/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-beta

HOW TO PUSH TO RELEASE

To push to release the following directories need to be pushsnip'ed on aus2-staging:

# contains every locale except 'sl'
/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU
# only contains 'sl'
/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only