ReleaseEngineering/How To/Generate partial updates: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 173: Line 173:


= Push to mirrors =
= Push to mirrors =
If you haven't released yet there isn't anything to do here, because everything will get pushed by the automation when the go comes. If the release has shipped already you need to push the new partials out.
From a build slave:
ssh -oIdentityFile=~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org
rsync -nav --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
  --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=*.checksums.asc \
  --exclude=logs --exclude=jsshell* --exclude=*.zip --exclude=*.zip.asc --exclude=contrib* \
  /pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/ /pub/mozilla.org/firefox/releases/$NEWV/
Verify the list is correct - expect the new partial.mar's + asc files, the SUMS files and their asc's, and the directories containing them.
Then remove the n from -nav to push for real. This should take < 15 min to copy; allow 3 hours for mirror a reasonable number of mirrors to sync.


= Final verification =
= Final verification =


= Pushsnip =
= Pushsnip =

Revision as of 04:11, 12 July 2012

Intro

This document is about generating partial updates from older releases, eg N-2 to N, on request from Release Management. We generate N-1 to N updates in the release automation, but more than that is a manual process until bug 575317 makes some progress.

Assumptions:

  • the automated updates builder has already run, and you're re-using the slave for the copy of the utilities and scripts you need
  • the request is for the release channel
  • all locales are being generated, possibly only some of the platforms

For the purposes of this doc we'll say the update will be between $OLDV and $NEWV build$BUILD.

Create config files

Patcher config

To get setup

  • grab a copy of the release config, mozRelease-branch-patcher2.cfg, from your own CVS checkout or from mxr
  • verify that the file is up to date by checking the values of from and to in <current-update>, and the right buildN in the completemarurl of the <releases> block for the release being created
  • rename it to mozRelease-branch-patcher2-XXXXXX.cfg, where XXXXXX is the bug number for the request

To modify the config

  • in the <current-update> block
    • change the from version to $OLDV
    • in the <partial> block change the from version in the paths to $OLDV
  • remove all the past-update lines just below <current-update> block
  • remove the unneeded blocks from <release> (keep only $OLDV and $NEWV, for brevity)
  • if only some platforms are needed (eg windows), remove the other platforms from the <platforms> and <exceptions blocks

Attach the file to the bug, request review. You'll download it again later.

Update verify configs

To get setup

  • grab a copy of the configs, either from your tools checkout or directly from hg.m.o (windows, mac)
  • verify the configs are up to date by checking the current release is referenced in the 2nd line, and that the to definition there refers to the right buildN path
  • rename to mozRelease-firefox-win32-XXXXXX.cfg, where XXXXX is the bug number for the request

To modify the configs:

  • remove all the lines that aren't for $OLDV and the current shipped release
  • in $OLDV's full check line
    • add the locales from $OLDV's quick check line
    • add the patch_types, aus_server, ftp_server, and to definitions from the current release
  • remove the $OLDV quick check comment and config line
  • remove current release lines
  • repeat for each platform

Attach the files to the bug as a patch (eg via mq), request review. You'll download it again later.

Generate the partials

Initial setup (~15 min) 

cd /builds/slave/rel-m-rel-updates/build
mv temp/ temp.automated_run/
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=YYYYYY \
  > patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg
# where YYYYYY is the attachment number for the patcher config
#   and XXXXXX is the bug number

# get source and build tools - a checksum based run doesn't have these
# NB: not UPDATE_PACKAGING_R16, we need a more recent version of mar
# use $NEWV's tag, eg FIREFOX_13_0_1_RELEASE
export HGROOT=http://hg.mozilla.org/releases/mozilla-release
perl patcher2.pl --build-tools-hg --tools-revision=FIREFOX_VER_RELEASE \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Get the complete mar files (~45 mins for windows & mac) 

perl patcher2.pl --download \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Generate unsigned partial updates (90 mins??) 

perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg \
 --app=firefox --brand=Firefox \
 --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Make a backup of the new partials/snippets, destroy snippets (< 5 min) 

rsync -a temp/firefox/$OLDV-$NEWV{,-unsigned}/
rm -rf temp/firefox/$OLDV-$NEWV/aus2*

Sign the mar files (< 15 min ??) 

# get a token
# look up the values for --user in the release-signing part of passwords.py, on the master
unset HISTFILE
IP=`/sbin/ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk '{print $1}'`
ssh -oBatchMode=no cltbld@buildbot-master12.build.mozilla.org \
  'curl --fail -k -Fslave_ip=$IP -Fduration=3600 --user reda:cted \
   https://signing1.build.scl1.mozilla.com:9120/token' \
     > /builds/slave/rel-m-rel-updates/token
rm /builds/slave/rel-m-rel-updates/nonce

# update the channel id and version, then sign
for f in `find temp/firefox/$OLDV-$NEWV/ftp -type f -name *partial.mar | sort`; do
  echo Signing $f
  mozilla/obj/dist/host/bin/mar -H firefox-mozilla-release -V $NEWV -i $f;
  /tools/python-2.6.5/bin/python \
    /builds/slave/rel-m-rel-updates/tools/release/signing/signtool.py \
    -t /builds/slave/rel-m-rel-updates/token -n /builds/slave/rel-m-rel-updates/nonce \
    -c /builds/slave/rel-m-rel-updates/tools/release/signing/host.cert \
    -H signing1.build.scl1.mozilla.com:9120 -H signing2.build.scl1.mozilla.com:9120 -f gpg -f mar \
    "$f"
done

Regenerate the snippets (~ 1 min) 

perl patcher2.pl --create-patchinfo \
  --app=firefox --brand=Firefox \
  --config=patcher-configs/mozRelease-branch-patcher2-XXXXXX.cfg

Snippet verification (< 1 min) 

# release==releasetest ?
cd temp/firefox/$OLDV-$NEWV
bash /builds/slave/rel-m-rel-updates/tools/release/compare-channel-snippets.sh \
  aus2 release aus2.test  releasetest

Tidy and upload mar files (~5 min ??) 

find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type f -exec chmod 644 {} \; 
find ftp/firefox/nightly/$NEWV-candidates/build$BUILD -type d -exec chmod 755 {} \;
# $BUILD is the build number, eg 1, 2, ...
rsync -nav -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' \
  ftp/firefox/nightly/$NEWV-candidates/build$BUILD/update \
  ffxbld@stage.mozilla.org:/pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/

Upload snippets (~ 1 min) 

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2.test/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial-test

rsync -av -e 'ssh -oIdentityFile=~/.ssh/auspush' --exclude=complete.txt \
 aus2/ \
 ffxbld@aus3-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-$NEWV-build$BUILD-$OLDV-partial

Backupsnip/pushsnip the test snippets 

ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/backupsnip Firefox-$NEWV-build$BUILD-$OLDV-partial-test
ssh -oIdentityFile=~/.ssh/auspush ffxbld@aus3-staging.mozilla.org \
  bin/pushsnip   Firefox-$NEWV-build$BUILD-$OLDV-partial-test

Announce to r-d so QA can test/include in testing.

Update Verify

Do our full old + update == new checks.

On each platform that needs testing

  • stop buildbot slave
  • in a terminal (SSH is OK for linux & mac)
# windows
cd /e/builds/slave
# else
cd /builds/slave

# all platforms
mkdir bugXXXXXX; cd $_
hg clone http://hg.mozilla.org/build/tools scripts
hg -R scripts update -C -r FIREFOX_13_0_1_RELEASE_RUNTIME

cd scripts/release/updates/
# ZZZZZZ is the attachment number for the verify config
wget -O- https://bugzilla.mozilla.org/attachment.cgi?id=ZZZZZZ | patch -p3

# mac
bash verify.sh -c mozRelease-firefox-mac64-XXXXXX.cfg 2>&1 |tee uv.log; echo $?
# win32
bash verify.sh -c mozRelease-firefox-win32-XXXXXX.cfg 2>&1 |tee uv.log; echo $?

Verify exit code is 0, reboot slave to get back in service.

Configure bouncer

Add a product: Firefox-$NEWV-Partial-$OLDV Add locations: 

osx  /firefox/releases/$NEWV/update/mac/:lang/firefox-$OLDV-$NEWV.partial.mar
win  /firefox/releases/$NEWV/update/win32/:lang/firefox-$OLDV-$NEWV.partial.mar

Redo SUMS files

Push to mirrors

If you haven't released yet there isn't anything to do here, because everything will get pushed by the automation when the go comes. If the release has shipped already you need to push the new partials out.

From a build slave: 

ssh -oIdentityFile=~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org
rsync -nav --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* \
 --exclude=*update-backup* --exclude=*partner-repacks* --exclude=*.checksums --exclude=*.checksums.asc \
 --exclude=logs --exclude=jsshell* --exclude=*.zip --exclude=*.zip.asc --exclude=contrib* \
 /pub/mozilla.org/firefox/nightly/$NEWV-candidates/build$BUILD/ /pub/mozilla.org/firefox/releases/$NEWV/

Verify the list is correct - expect the new partial.mar's + asc files, the SUMS files and their asc's, and the directories containing them.

Then remove the n from -nav to push for real. This should take < 15 min to copy; allow 3 hours for mirror a reasonable number of mirrors to sync.

Final verification

Pushsnip

Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Generate_partial_updates&oldid=449982"