ReleaseEngineering/How To/Generate partial updates

Intro

This document is about generating partial updates from older releases, eg N-2 to N, on request from Release Management. We generate N-1 to N updates in the release automation, but more than that is a manual process until bug 575317 makes some progress.

Assumptions:

• the automated updates builder has already run, and you're re-using the slave for the copy of the utilities and scripts you need
• the request is for the release channel
• all locales are being generated, possibly only some of the platforms

For the purposes of this doc we'll say the update will be between $OLDV and $NEWV build$BUILD. The bug is $XXXXXX

Create config files

Patcher config

To get setup

• grab a copy of the release config, mozRelease-branch-patcher2.cfg, from your own CVS checkout or from mxr
• verify that the file is up to date by checking the values of from and to in <current-update>, and the right buildN in the completemarurl of the <releases> block for the release being created
• rename it to mozRelease-branch-patcher2-$XXXXXX.cfg

To modify the config

• in the <current-update> block
  • change the from version to $OLDV
  • in the <partial> block change the from version in the paths to $OLDV
• remove all the past-update lines just below <current-update> block
• remove the unneeded blocks from <release> (keep only $OLDV and $NEWV, for brevity)
• if only some platforms are needed (eg windows), remove the other platforms from the <platforms> and <exceptions blocks

Attach the file to the bug, request review. You'll download it again later using $YYYYYY for the attachment id.

Update verify configs

To get setup

• grab a copy of the configs, either from your tools checkout or directly from hg.m.o (windows, mac)
• verify the configs are up to date by checking the current release is referenced in the 2nd line, and that the to definition there refers to the right buildN path
• rename to mozRelease-firefox-win32-$XXXXXX.cfg

To modify the configs:

• remove all the lines that aren't for $OLDV and the current shipped release
• in $OLDV's full check line
  • add the locales from $OLDV's quick check line
  • add the patch_types, aus_server, ftp_server, and to definitions from the current release
• remove the $OLDV quick check comment and config line
• remove current release lines
• repeat for each platform

Attach the files to the bug as a patch (eg via mq), request review. You'll download it again later using $ZZZZZZ as the attachment id.

Generate the partials

Initial setup (~15 min)

cd /builds/slave/rel-m-rel-updates/build
mv temp/ temp.automated_run/
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=$YYYYYY \
  > patcher-configs/mozRelease-branch-patcher2-$XXXXXX.cfg
# where YYYYYY is the attachment number for the patcher config

# get source and build tools - a checksum based run doesn't have these
# NB: not UPDATE_PACKAGING_R16, we need a more recent version of mar
# use $NEWV's tag, eg FIREFOX_13_0_1_RELEASE
export HGROOT=http://hg.mozilla.org/releases/mozilla-release
perl patcher2.pl --build-tools-hg --tools-revision=FIREFOX_VER_RELEASE \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-$XXXXXX.cfg

Get the complete mar files (~45 mins for windows & mac)

perl patcher2.pl --download \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-$XXXXXX.cfg

Generate unsigned partial updates (90 mins??)

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg \
 --app=firefox --brand=Firefox \
 --config=patcher-configs/mozRelease-branch-patcher2-$XXXXXX.cfg 

Make a backup of the new partials/snippets, destroy snippets (< 5 min)

rsync -a temp/firefox/$OLDV-$NEWV{,-unsigned}/
rm -rf temp/firefox/$OLDV-$NEWV/aus2*

Sign the mar files (< 15 min ??)

# get a token
# look up the values for --user in the release-signing part of passwords.py, on the master
unset HISTFILE
IP=`/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`
ssh -oBatchMode=no cltbld@buildbot-master12.build.mozilla.org \
  'curl --fail -k -Fslave_ip=$IP -Fduration=3600 --user reda:cted \
   https://signing1.build.scl1.mozilla.com:9120/token' \
     > /builds/slave/rel-m-rel-updates/token
rm /builds/slave/rel-m-rel-updates/nonce

# update the channel id and version, then sign
for f in `find temp/firefox/$OLDV-$NEWV/ftp -type f -name *partial.mar | sort`; do
  echo Signing $f
  mozilla/obj/dist/host/bin/mar -H firefox-mozilla-release -V $NEWV -i $f;
  /tools/python-2.6.5/bin/python \
    /builds/slave/rel-m-rel-updates/tools/release/signing/signtool.py \
    -t /builds/slave/rel-m-rel-updates/token -n /builds/slave/rel-m-rel-updates/nonce \
    -c /builds/slave/rel-m-rel-updates/tools/release/signing/host.cert \
    -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar \
    "$f"
done

Regenerate the snippets (~ 1 min)

perl patcher2.pl --create-patchinfo \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-$XXXXXX.cfg

Snippet verification (< 1 min)

# release==releasetest ?
cd temp/firefox/$OLDV-$NEWV
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh \
  aus2 release aus2.test  releasetest

Tidy and upload mar files (~5 min ??)

find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type f -exec chmod 644 {} \; 
find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type d -exec chmod 755 {} \;
# $BUILD is the build number, eg 1, 2, ...
rsync -nav -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' \
  ftp/firefox/nightly/$NEWV-candidates/build$BUILD/update \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/

Upload snippets (~ 1 min)

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2.test/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial-test

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial

Backupsnip/pushsnip the test snippets

ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/backupsnip Firefox-$NEWV-build$BUILD-$OLDV-partial-test
ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/pushsnip   Firefox-$NEWV-build$BUILD-$OLDV-partial-test

Announce to r-d so QA can test/include in testing.

Update Verify

Do our full old + update == new checks.

On each platform that needs testing

• stop buildbot slave
• in a terminal (SSH is OK for linux & mac)

# windows
cd /e/builds/slave
# else
cd /builds/slave

# all platforms
mkdir bug$XXXXXX; cd $_
hg clone http://hg.mozilla.org/build/tools scripts
hg -R scripts update -C -r FIREFOX_13_0_1_RELEASE_RUNTIME

cd scripts/release/updates/
# ZZZZZZ is the attachment number for the verify config
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=$ZZZZZZ | patch -p3

# mac
bash verify.sh -c mozRelease-firefox-mac64-$XXXXXX.cfg 2>&1 |tee uv.log; echo $?
# win32
bash verify.sh -c mozRelease-firefox-win32-$XXXXXX.cfg 2>&1 |tee uv.log; echo $?

Verify exit code is 0, reboot slave to get back in service.

Configure bouncer

Add a product: Firefox-$NEWV-Partial-$OLDV Add locations:

osx  /firefox/releases/$NEWV/update/mac/:lang/firefox-$OLDV-$NEWV.partial.mar
win  /firefox/releases/$NEWV/update/win32/:lang/firefox-$OLDV-$NEWV.partial.mar

Redo SUMS files

... so that they include the new partials.

# ffxbld@stage
# backup in case of disaster, adjust platforms as required
rsync -av --include=*SUMS* \
 --include=win32 --include=win32/* --include=win32/*/*checksums \
 --include=mac --include=mac/* --include=mac/*/*checksums \
 --exclude=* \
 /pub/mozilla.org/firefox/candidates/$NEWV-candidates/build$BUILD/ \
 /tmp/bug$XXXXXX/

cd /pub/mozilla.org/firefox/candidates/$NEWV-candidates/build$BUILD/update
for f in `find -name firefox-$OLDV-$NEWV.partial.mar`; do
  csfile=../`dirname $f`/firefox-$NEWV.checksums
  echo $csfile
  for hash in sha512 md5 sha1; do
     echo `openssl dgst -$hash $f | cut -d' ' -f2` $hash `stat -c '%s' $f` update/$f >> $csfile
  done
done

Then force build the release-mozilla-release-firefox_checksums builder on the buildbot master. If that fails there are uber-painful manual steps in here.

Push to mirrors

If you haven't released yet there isn't anything to do here, because everything will get pushed by the automation when the go comes. If the release has shipped already you need to push the new partials out.

From a build slave:

ssh -oIdentityFile=~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org
rsync -nav --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
 --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=*.checksums.asc \
 --exclude=logs --exclude=jsshell* --exclude=*.zip --exclude=*.zip.asc --exclude=contrib* \
 /pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/ /pub/mozilla.org/firefox/releases/$NEWV/

Verify the list is correct - expect the new partial.mar's + asc files, the SUMS files and their asc's, and the directories containing them.

Then remove the n from -nav to push for real. This should take < 15 min to copy info firefox/releases/; allow 3 hours for a reasonable number of mirrors to rsync it.

Final verification

Verify bouncer is working OK and mirrors are responding properly.

On a linux slave

mkdir /builds/slave/bug$XXXXXX/; cd $_
hg clone http://hg.mozilla.org/build/tools
# eg FIREFOX_13_0_1_RELEASE_RUNTIME
hg -R tools update -C -r FIREFOX_VER_RELEASE_RUNTIME
cd tools/release/updates
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=$ZZZZZZ | patch -p3
cd ..

# adjust platforms as necessary
bash final-verification.sh mozRelease-firefox-{win32,mac64}-$XXXXXX.cfg 2>&1 |tee log

Check for errors, eg 404's.

Pushsnip

Once RelMan gives the go

# ffxbld@aus3-staging
backupsnip Firefox-$NEWV-build$BUILD-$OLDV-partial
pushsnip  Firefox-$NEWV-build$BUILD-$OLDV-partial

Both operations should be quick because they only operate on a single version.

This will be done after the main push of Firefox-$NEWV-build$BUILD if the partials where generated prior to release.