Release Engineers

Nick Thomas

Bugs

bug 464351 - tracking bug
bug 461914 - null partner updates

Generate Major Update Snippets

These are for the "vanilla" builds we're going to update.

# ssh cltbld@prometheus-vm.build.mozilla.org
mkdir -p /builds/2.0.0.18-major-update/snippets
cd /builds/2.0.0.18-major-update/snippets
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R5 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R5 -d MozBuild mozilla/tools/release/MozBuild
cd ../
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs

cd patcher
export CVSROOT=:ext:ffxbld@cvs.mozilla.org:/cvsroot

# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R5 --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.0.4 2.0.0.18-3.0.4
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Separate out beta snippets

cd temp/firefox/2.0.0.18-3.0.4
rsync -a aus2/ aus2.beta/
find aus2.beta/ -type d -name "release" -exec rm -rfv {} \;
find aus2/ -type d -name "beta" -exec rm -rfv {} \; 

Quick verify

Check the snippets are identical for all four channels

# betatest == releasetest
find aus2.test -type d -iregex '.*betatest.*' | perl -nle '$a = $_; $a =~ s/betatest/releasetest/; system("diff -r -u $_ $a");'
# test the opposite direction to catch anything that only exists for releasetest
find aus2.test -type d -iregex '.*releasetest.*' | perl -nle '$a = $_; $a =~ s/releasetest/betatest/; system("diff -r -u $_ $a");'

# betatest == beta
find aus2.test -type d -iregex '.*betatest.*' | perl -nle '$a = $_; $a =~ s/betatest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/betatest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");' 

# betatest == release
find aus2.test -type d -iregex '.*betatest.*' | perl -nle '$a = $_; $a =~ s/betatest/release/; $a =~ s/aus2\.test/aus2/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/betatest/; $a =~ s/aus2/aus2.test/; system("diff -r -u $_ $a");'

Push to AUS

cd /builds/2.0.0.18-major-update/snippets/patcher/temp/firefox/2.0.0.18-3.0.4
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081117-Firefox-2.0.0.18-3.0.4-MU
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081117-Firefox-2.0.0.18-3.0.4-MU-beta
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081117-Firefox-2.0.0.18-3.0.4-MU-test

Enable test snippets

Safe to do this without doing null snippets because we're only talking about betatest and releasetest, where no end-users of partner builds will be falling back to (metrics website confirms).

~/bin/backupsnip 20081117-Firefox-2.0.0.18-3.0.4-MU-test
~/bin/pushsnip 20081117-Firefox-2.0.0.18-3.0.4-MU-test

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-20018-304-real-major/
cd /builds/verify/firefox-20018-304-real-major/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d updates mozilla/testing/release/updates/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/

# get patch to trawl "Only in" directories
cd common
# on mac & linux
curl -s https://bugzilla.mozilla.org/attachment.cgi?id=304252 | patch -p0
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=304252
patch -p0 < patch

# everyone ...
platform=linux   # or mac or win32
cd ../updates
./verify.sh -c moz18-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Generate Null Partner Snippets

Create snippets

Starting from attachment 349718, we create empty snippets to block partner builds from updating.

# ssh cltbld@prometheus-vm.build.mozilla.org
cd /builds/2.0.0.18-major-update/snippets/patcher/
# get a copy of https://bugzilla.mozilla.org/attachment.cgi?id=349718 ...
#    and save as partner-update-channels
# checkout out null_partners.sh and update it
cd temp/firefox/2.0.0.18-3.0.4
bash ../../../one-offs/null_partners.sh ../../../partner-update-channels

NB: just going to do the release-* snippets this time, since there are no users on releastest, beta, betatest. Now check there are no vanilla snippets in there [no extended regexp in find :-(]

find aus2.null -regex '.*/release/.*'
find aus2.null -regex '.*/releasetest/.*'
find aus2.null -regex '.*/beta/.*'
find aus2.null -regex '.*/betatest/.*'

Push to AUS server

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.null/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20081124-Firefox-2.0.0.18-3.0.4-MU-partner-null

Push live

13780 files in all

# cltbld@aus2-staging
~/bin/backupsnip 20081124-Firefox-2.0.0.18-3.0.4-MU-partner-null
~/bin/pushsnip 20081124-Firefox-2.0.0.18-3.0.4-MU-partner-null

Verify nulls

Create a list of URLs to test

cd /builds/2.0.0.18-major-update/snippets/one-offs
cp create_aus_urls.sh create_aus_urls-win32.sh
nano aus_urls-win32.sh

and change the version, BuildID, leave only win32 in the platform list. Also set it to read in from partner-update-channels, and write to platform specific file. Repeat for linux, mac.

We create a null.xml for comparison

wget -O null.xml https://aus2.mozilla.org/update/1/Firefox/2.0.0.18/2008102918/WINNT_x86-msvc/af/release-cck-acer-google/update.xml

Generate platform-specific locale lists

ls temp/firefox/2.0.0.18-3.0.4/aus2/Firefox/2.0.0.18/WINNT_x86-msvc/2008102918/ > locales-win32.txt
one-offs/create_aus_urls-win32.sh
ls temp/firefox/2.0.0.18-3.0.4/aus2/Firefox/2.0.0.18/Darwin_Universal-gcc3/2008102919/ > locales-mac.txt
ls temp/firefox/2.0.0.18-3.0.4/aus2/Firefox/2.0.0.18/Linux_x86-gcc3/2008102919/ > locales-linux.txt

Run tests for win32

one-offs/test_null_updates.sh urls-win32.txt 2>&1 | tee test-nulls-win32.txt
grep -v '^PASS' test-nulls-win32.txt
FAIL https://aus2.mozilla.org/update/1/Firefox/2.0.0.18/2008102918/WINNT_x86-msvc/de/release-cck-tonline/update.xml

Which is expected, because we already enabled MU for T-Online and it's only the de locale. Same again for each other platform.

Uploaded files used/created to bug 461914.

Enable beta snippets

Safe to do this without doing null snippets because there are no end-users of partner builds on beta channel (kev and metrics website confirmed).

cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20081117-Firefox-2.0.0.18-3.0.4-MU-beta
real    27m28.805s
user    0m28.542s
sys     0m30.046s
time ~/bin/pushsnip 20081117-Firefox-2.0.0.18-3.0.4-MU-beta
real    0m7.887s
user    0m0.006s
sys     0m0.401s

Enable release snippets

• In case you need to cancel/abort updates, the quickest way to do this is here
• Ahead of time

# cltbld@aus2-staging
time ~/bin/backupsnip 20081117-Firefox-2.0.0.18-3.0.4-MU
 real    32m35.668s
 user    0m31.077s
 sys     0m56.322s

• Actual release

# cltbld@aus2-staging
time ~/bin/pushsnip 20081117-Firefox-2.0.0.18-3.0.4-MU
 real    0m5.623s
 user    0m0.008s
 sys     0m0.418s

Remove nulls

Once Fx 2.0.0.19 ships, we should remove the nulls for partners that have not had a major update yet. Will do this in bug 461914.