ReleaseEngineering/How To/Generate partial updates

Intro

This document is about generating partial updates from older releases, eg N-2 to N, on request from Release Management. We generate N-1 to N updates in the release automation, but more than that is a manual process until bug 575317 makes some progress.

Assumptions:

• the automated updates builder has already run, and you're re-using the slave for the copy of the utilities and scripts you need
• the request is for the release channel
• all locales are being generated, possibly only some of the platforms

For the purposes of this doc we'll say the update will be between $OLDV and $NEWV build$BUILD.

Config files

Patcher config

To get setup

• grab a copy of the release config, mozRelease-branch-patcher2.cfg, from your own CVS checkout or from mxr
• verify that the file is up to date by checking the values of from and to in <current-update>, and the right buildN in the completemarurl of the <releases> block for the release being created
• rename it to mozRelease-branch-patcher2-XXXXXX.cfg, where XXXXXX is the bug number for the request

To modify the config

• in the <current-update> block
  • change the from version to $OLDV
  • in the <partial> block change the from version in the paths to $OLDV
• remove all the past-update lines just below <current-update> block
• remove the unneeded blocks from <release> (keep only $OLDV and $NEWV, for brevity)
• if only some platforms are needed (eg windows), remove the other platforms from the <platforms> and <exceptions blocks

Attach the file to the bug, request review. You'll download it again later.

Update verify configs

To get setup

• grab a copy of the configs, either from your tools checkout or directly from hg.m.o (windows, mac)
• verify the configs are up to date by checking the current release is referenced in the 2nd line, and that the to definition there refers to the right buildN path
• rename to mozRelease-firefox-win32-XXXXXX.cfg, where XXXXX is the bug number for the request

To modify the configs:

• remove all the lines that aren't for $OLDV and the current shipped release
• in $OLDV's full check line
  • add the locales from $OLDV's quick check line
  • add the patch_types, aus_server, ftp_server, and to definitions from the current release
• remove the $OLDV quick check comment and config line
• remove current release lines
• repeat for each platform

Attach the files to the bug as a patch (eg via mq), request review. You'll download it again later.

Generate the partials

Initial setup (~15 min)

cd /builds/slave/rel-m-rel-updates/build
mv temp/ temp.automated_run/
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=YYYYYY \
  > patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg
# where YYYYYY is the attachment number for the patcher config
#   and XXXXXX is the bug number

# get source and build tools - a checksum based run doesn't have these
# NB: not UPDATE_PACKAGING_R16, we need a more recent version of mar
# use $NEWV's tag, eg FIREFOX_13_0_1_RELEASE
export HGROOT=http://hg.mozilla.org/releases/mozilla-release
perl patcher2.pl --build-tools-hg --tools-revision=FIREFOX_VER_RELEASE \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Get the complete mar files (~45 mins for windows & mac)

perl patcher2.pl --download \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Generate unsigned partial updates (90 mins??)

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg \
 --app=firefox --brand=Firefox \
 --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg 

Make a backup of the new partials/snippets, destroy snippets (< 5 min)

rsync -a temp/firefox/$OLDV-$NEWV{,-unsigned}/
rm -rf temp/firefox/$OLDV-$NEWV/aus2*

Sign the mar files (< 15 min ??)

# get a token
# look up the values for --user in the release-signing part of passwords.py, on the master
unset HISTFILE
IP=`/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`
ssh -oBatchMode=no cltbld@buildbot-master12.build.mozilla.org \
  'curl --fail -k -Fslave_ip=$IP -Fduration=3600 --user reda:cted \
   https://signing1.build.scl1.mozilla.com:9120/token' \
     > /builds/slave/rel-m-rel-updates/token
rm /builds/slave/rel-m-rel-updates/nonce

# update the channel id and version, then sign
for f in `find temp/firefox/$OLDV-$NEWV/ftp -type f -name *partial.mar | sort`; do
  echo Signing $f
  mozilla/obj/dist/host/bin/mar -H firefox-mozilla-release -V $NEWV -i $f;
  /tools/python-2.6.5/bin/python \
    /builds/slave/rel-m-rel-updates/tools/release/signing/signtool.py \
    -t /builds/slave/rel-m-rel-updates/token -n /builds/slave/rel-m-rel-updates/nonce \
    -c /builds/slave/rel-m-rel-updates/tools/release/signing/host.cert \
    -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar \
    "$f"
done

Regenerate the snippets (~ 1 min)

perl patcher2.pl --create-patchinfo \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Snippet verification (< 1 min)

# release==releasetest ?
cd temp/firefox/$OLDV-$NEWV
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh \
  aus2 release aus2.test  releasetest

Tidy and upload mar files (~5 min ??)

find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type f -exec chmod 644 {} \; 
find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type d -exec chmod 755 {} \;
# $BUILD is the build number, eg 1, 2, ...
rsync -nav -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' \
  ftp/firefox/nightly/$NEWV-candidates/build$BUILD/update \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/

Upload snippets (~ 1 min)

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2.test/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial-test

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial

Backupsnip/pushsnip the test snippets

ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/backupsnip Firefox-$NEWV-build$BUILD-$OLDV-partial-test
ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/pushsnip   Firefox-$NEWV-build$BUILD-$OLDV-partial-test

Announce to r-d so QA can test/include in testing.