ReleaseEngineering/Unified Release Process

Warning: This RelEng page is obsolete!This is the ancient release process for Firefox and Thunderbird. It's since been replaced with the new release automation

1 Release prep
- 1.1 Create sign-off sheet
- 1.2 Configure each build
2 Bump version number
3 Tag the release
4 Build en-US
5 Archive the source
6 Build l10n
7 Signing windows files
8 Repackaging
- 8.1 Verify partner repacks
- 8.2 Partner repack QA tests
9 Updates
- 9.1 Update verification
10 Stage release
11 Sign builds
12 Merge files
- 12.1 Merge update (MAR) files
- 12.2 Merge signed files
13 Update bouncer links
14 Push the button
- 14.1 Make the release live
- 14.2 Make the update live
15 Reset for next release
- 15.1 Bump version number

Release prep

Create sign-off sheet

Seek l10n and code freeze signoff (including bonsai queries)

https://intranet.mozilla.org/Release:Template:Product

Copy to ${PRODUCT}-${VERSION} and link from https://intranet.mozilla.org/Release

Update config and copy into shell before running commands.

Configure each build

Edit the tree's tinder-config.pl settings. These are in CVS in /cvsroot/mozilla/tools/tinderbox-configs/ on the ${BRANCH_TAG}_release

$UseTimeStamp should be off
Set BuildTag to ${PRODUCT_TAG}_RELEASE
Set milestone to ${PRODUCT}${VERSION}
$update_pushinfo should be 0
$update_package should be 1

File a bug with the changes and request review.

Tag with ${PRODUCT_TAG}_RELEASE and ${PRODUCT_TAG}_RC${RC} after checkin.

Bump version number

Three files need to be updated to the release version number (removing "pre") in cvsroot/mozilla/ :

${APP}/app/module.ver
${APP}/config/version.txt
config/milestone.txt

These files should be changed on a minibranch. On the tagging machine (karma.build.mozilla.org):

cvs -d :ext:cltbld@cvs.mozilla.org:/cvsroot co -r ${BRANCH_TAG}\
 mozilla/${APP}/app/module.ver mozilla/${APP}/config/version.txt \
 mozilla/config/milestone.txt

cd mozilla/

# remove "pre" from all version numbers
vi ${APP}/app/module.ver ${APP}/config/version.txt config/milestone.txt

# create minibranch
cvs tag -b ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \
 ${APP}/config/version.txt config/milestone.txt
cvs up -r ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \
 ${APP}/config/version.txt config/milestone.txt
cvs ci -m"bumping version number on ${APP} on minibranch only"

Tag the release

Review the Tinderbox tree: http://tinderbox.mozilla.org/showbuilds.cgi?tree=$TBOX_TREE

We use karma (a build system) as a scratch area to tag repository files.

Set up the system to tag

# Log into karma as cltbld.
# use "screen" to preserve session 
screen
# Create the main tag directory.
cd ~cltbld
mkdir -p tags/${PRODUCT_TAG}_RC${RC}
cd tags && ln -s ${PRODUCT_TAG}_RC${RC} ${PRODUCT_TAG}_RELEASE

Tagging cvsroot/mozilla

# Create the cvsroot tag directory.
cd ~cltbld/tags/${PRODUCT_TAG}_RC${RC}
mkdir cvsroot/
cd cvsroot/

# Check out Mozilla from the branch you want to tag.
cvs co \
  -r $BRANCH_TAG \
  -D "$PULL_DATE" \
  mozilla/client.mk 

cd mozilla
gmake -f client.mk checkout \
  MOZ_CO_PROJECT=all \
  MOZ_CO_DATE="$PULL_DATE" \
  2>&1 | tee ../make_checkout.log

# create minibranch if its not already there
cvs tag -b ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \
 ${APP}/config/version.txt config/milestone.txt
# pull version files from the version bump minibranch
cvs up -r ${PRODUCT_TAG}_MINIBRANCH ${APP}/app/module.ver \
 ${APP}/config/version.txt config/milestone.txt

# Create the tag.
cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log
# Look for errors
grep -v "^T" ../${PRODUCT_TAG}_RELEASE.log | grep -v '^cvs tag'

# Create the RC tag.
cvs tag ${PRODUCT_TAG}_RC${RC} 2>&1 | tee ../${PRODUCT_TAG}_RC${RC}.log
# Look for errors
grep -v "^T" ../${PRODUCT_TAG}_RC${RC}.log | grep -v '^cvs tag'

# Create a minibranch for the pull scripts so we can change them without
# changing anything on the original branch.
cvs tag -b ${PRODUCT_TAG}_MINIBRANCH client.mk

# Update client.mk to the minibranch you just created.
cvs up -r ${PRODUCT_TAG}_MINIBRANCH client.mk

# Add the new product tag to the client.mk
sed "s/${BRANCH_TAG}/${PRODUCT_TAG}_RELEASE/g" client.mk > client.mk.tmp
mv client.mk.tmp client.mk

# Verify that client.mk only has the changes you expect
cvs diff -u client.mk
# Verify that client.mk has its sticky tag set properly.
cvs status client.mk
cvs commit -m "For $PRODUCT $VERSION, redirect client.mk onto the ${PRODUCT_TAG}_RELEASE tag." client.mk

# Move the release tag onto the modified version of the pull scripts.
cvs tag -F ${PRODUCT_TAG}_RELEASE client.mk
cvs tag -F ${PRODUCT_TAG}_RC${RC} client.mk

Tagging mofo/talkback/fullsoft

# Create the mofo tag directory.
cd ~cltbld/tags/${PRODUCT_TAG}_RC${RC}
mkdir mofo/
cd mofo/

# Check out the talkback files from the branch you want to tag.
cvs -d :ext:cltbld@cvs.mozilla.org:/mofo co \
  -r $BRANCH_TAG \
  -D "$PULL_DATE" \
  talkback/fullsoft 2>&1 | tee checkout-output.log

# Create the tag.
cd talkback/fullsoft/
cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log

# Look for errors
grep -v "^T" ../${PRODUCT_TAG}_RELEASE.log | grep -v '^cvs tag'

Tagging l10n/l10n

# Create the l10n tag directory.
cd ~cltbld/tags/${PRODUCT_TAG}_RELEASE
mkdir l10n/
cd l10n/

# Grab list of shipped locales
LOCALES=`awk '{print $1}' < ../cvsroot/mozilla/${APP}/locales/shipped-locales`

# Check out the l10n files from the branch you want to tag.
(for l in $LOCALES
do
  cvs -d :ext:cltbld@cvs.mozilla.org:/l10n co \
    -r $BRANCH_TAG \
    -D "$l10n_PULL_DATE" \
    l10n/${l}
done) 2>&1 | tee checkout-output.log

cd l10n/

# Create the release tag.
cvs tag ${PRODUCT_TAG}_RELEASE 2>&1 | tee ../${PRODUCT_TAG}_RELEASE.log
 # Look for errors
grep -v '^T' ../${PRODUCT_TAG}_RELEASE.log

# Create the RC tag.
cvs tag ${PRODUCT_TAG}_RC${RC} 2>&1 | tee ../${PRODUCT_TAG}_RC${RC}.log
# Look for errors
grep -v '^T' ../${PRODUCT_TAG}_RC${RC}.log

Update repository information

Update http://developer.mozilla.org/en/docs/CVS_Tags with the time & date of the tag.

Build en-US

Identify the builds that will be used for the release.

Build

Remove the tree's last-built file.

For each platform, identify build machine.

On each build machine, log in as cltbld and run the following command in /builds/tinderbox/$BUILD_DIR :

 ./build-seamonkey.pl --once --mozconfig mozconfig --depend \
   --config-cvsup-dir `pwd`/tinderbox-configs \
   2>&1 | tee ${PRODUCT_TAG}-RC${RC}.log

Annotate ("star") the build on the tinderbox page.

Verify build logs

As each build completes, the tinderbox cell representing that build should turn green. Click on the "L" link, then "View full log" and verify that there are no fatal errors, and that the build ends with builds being copied to the appropriate location on stage.

Copy en-US builds to candidates directory

As each build completes, copy to candidates directory.

Review http://tinderbox.mozilla.org/showbuilds.cgi?tree=$TBOX_TREE
- Identify the directory the release build was pushed to (follow "D" link on desired build)
- Review logs ("L" link), there should be no errors.
Set PUSH_DATE to YYYY-MM-DD-HH in sign-off sheet.
Collect the builds together into a directory named "${VERSION}-candidates" under /home/ftp/pub/${PRODUCT}/nightly/.

#ssh to stage.mozilla.org as cltbld
cd /home/ftp/pub/$PRODUCT/nightly/
mkdir -p $VERSION-candidates/rc${RC}
rsync -av ${PUSH_DATE}-${PRODUCT}${VERSION}/ ${VERSION}-candidates/rc${RC}

Notify release team of availability.

Archive the source

Build the source tarballs. This should be performed on stage as the cltbld user.

NOTE - cltbld@stage:~/bin/ is a checkout of mofo/release/stage

cd ~/
mkdir -p ${PRODUCT}-${VERSION}/batch-source/rc${RC}
cd ${PRODUCT}-${VERSION}/batch-source/rc${RC}
~/bin/${PRODUCT}-src-tarball-nobuild -r ${PRODUCT_TAG}_RELEASE -m $VERSION 2>&1 | tee ${PRODUCT_TAG}_RC${RC}.log
chmod 644 *.bz2
rsync -n -av *.bz2 /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}
# verify that it only copies the source tarball
rsync -av *.bz2 /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}

Build l10n

Set up the l10n build systems to repackage the en-US files.

Configure each build

Edit the tree's tinder-config.pl settings. These are in CVS in /cvsroot/mozilla/tools/tinderbox-configs/ on the ${BRANCH_TAG}_l10n_release branch.

mozconfig:
Ensure that none of the mozconfigs define LOCALES_CO_DATE, only LOCALES_CO_TAG.
mk_add_options LOCALES_CO_TAG="${PRODUCT_TAG}_RELEASE"

tinder-config.pl:
Set %WGetFiles to the location of the en-US build.
$UseTimeStamp should be off
Set $BuildTag to ${PRODUCT_TAG}_RELEASE
Set $milestone to ${PRODUCT}${VERSION}

File a bug with the changes and request review.

Tag with ${PRODUCT_TAG}_RELEASE_l10n and ${PRODUCT_TAG}_RC${RC}_l10n after checkin.

Build

NOTE - this can be done without stopping the multi-tinderbox builds.

Remove the tree's last-built file.

For each platform, identify build machine.

On each build machine, log in as cltbld and run the following command in /builds/tinderbox/$BUILD_DIR :

 ./build-seamonkey.pl --once --mozconfig mozconfig --depend \
  --config-cvsup-dir `pwd`/tinderbox-configs \
  2>&1 | tee ${PRODUCT_TAG}-RC${RC}.log

Annotate ("star") the build on the tinderbox page.

Verify build logs

As each build completes, the tinderbox cell representing that build should turn green. Click on the "L" link, then "View full log" and verify that there are no fatal errors, and that the build ends with builds being copied to the appropriate location on stage.

Make a note of anywhere the word "busted" appears in the log next to a locale name; l10n builds attempt to repackage for all available locales, and will not turn red even if some or all of the locales fail to produce builds.

Minimum locales verification

Note - shipped-locales should be from the $BRANCH_TAG

Verify that we have every locale listed in shipped_locales:

for os in win32 linux mac
do
  ~/check_locales.sh /home/ftp/pub/${PRODUCT}/nightly/${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${os} \
~/${PRODUCT}-${VERSION}/batch-source/rc${RC}/mozilla/${APP}/locales/shipped-locales 
done

Output of "en-US not present" is normal at this stage.

Gather l10n files

Collect the l10n builds together into a directory named "${VERSION}-candidates" under /home/ftp/pub/${PRODUCT}/nightly/.

# ssh to stage.mozilla.org as cltbld
cd /home/ftp/pub/${PRODUCT}/nightly/
mkdir -p ${VERSION}-candidates/rc${RC}
rsync -n -av ${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${VERSION}-candidates/rc${RC}
# Verify that the output is what you expect!

rsync -av ${l10n_PUSH_DATE}-${PRODUCT}${VERSION}-l10n/ ${VERSION}-candidates/rc${RC}

l10n verification

Verify that l10n builds do not contain unexpected changes.

Run the l10n verification script on bm-xserve01 as cltbld (from VNC). The script is checked into /cvsroot/mozilla/testing/release/l10n/verify_l10n.sh

# convert version number
SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'`
SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'`

mkdir -p /builds/${PRODUCT}-${VERSION}-verification/l10n
cd /builds/${PRODUCT}-${VERSION}-verification/l10n
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d l10n mozilla/testing/release/l10n/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/
cd l10n
mkdir -p ${PRODUCT}-${SHORT_VERSION}-rc${RC}
# download current release
rsync --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \
-e ssh -av stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/ ${PRODUCT}-${SHORT_VERSION}-rc${RC}
./verify_l10n.sh ${PRODUCT}-${SHORT_VERSION}-rc${RC} 2>&1 | tee ${PRODUCT}-${SHORT_VERSION}-rc${RC}.log
# download the previous release
rsync --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \
-e ssh -Lav stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}
./verify_l10n.sh ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} 2>&1 | tee ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}.log
diff -r ${PRODUCT}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}/diffs \
${PRODUCT}-${SHORT_VERSION}-rc${RC}/diffs > l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt

# look for missing files or binary diffs
grep "^Only" l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt
grep "^Binary" l10n_metadiff_${SHORT_PREVIOUS_VERSION}_${SHORT_VERSION}-rc${RC}.txt

l10n QA tests

Send the metadiff and the location of the l10n builds to QA for testing.

Signing windows files

(installed bits, not the installer itself)

Follow these instructions.

Repackaging

TODO - clean up and merge distro repack process here

Create Google and Yahoo repackaged builds as per https://intranet.mozilla.org/Build:Repackaging:CJK_Repackaging

ssh cltbld@

argo-vm (linux)
bm-xserve01 (mac)
cerberus (win)

Download builds from stage

cd cjk-repackaging
mkdir original_builds-${VERSION}-rc${RC}
rsync -avn --include="*.ko*" --include="*.ja*" --include="*.zh*" --exclude="*" \
-e ssh stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/* \
original_builds-${VERSION}-rc${RC}

Repack

# NOTE - on windows this /cygdrive/c/cygwin/ *MUST* be before all paths
cd mozilla/extensions/brand-repackaging/
./repack.pl --path_to_original_builds=/home/cltbld/cjk-repackaging/original_builds-${VERSION}-rc${RC} \
--output_path_for_repacks=/home/cltbld/cjk-repackaging/repackaged_builds-${VERSION}-rc${RC}  \
--mar=/home/cltbld/cjk-repackaging/mar-tools/mozilla/dist/host/bin/mar 2>&1 | tee ${PRODUCT}-${VERSION}-RC${RC}.log

Copy to stage

ssh cltbld@stage.mozilla.org "mkdir -p ${PRODUCT}-${VERSION}/batch-cjk-repacks/original_builds"
cd ~/cjk-repackaging/repackaged_builds-${VERSION}-rc${RC}
rsync -av -e ssh * stage.mozilla.org:${PRODUCT}-${VERSION}/batch-cjk-repacks/original_builds/

Verify partner repacks

Verify that repacked builds do not contain unexpected changes.

Run the l10n verification script on bm-xserve01 as cltbld (from VNC) to produce a "metadiff". The script is checked into /cvsroot/mozilla/testing/release/l10n/verify_l10n.sh

# convert version number
SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'`
SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'`

mkdir -p /builds/${PRODUCT}-${VERSION}-verification/repacks
cd /builds/${PRODUCT}-${VERSION}-verification/repacks
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d l10n mozilla/testing/release/l10n/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/

cd /builds/${PRODUCT}-${VERSION}-verification/repacks/l10n
for partner in $PARTNERS
do
  mkdir -p ${partner}-${SHORT_VERSION}-rc${RC}
  rsync -av -e ssh --include="*.en-US.mac.dmg" --include="*en-US.win32.installer.exe" --include="*.en-US.linux-i686.tar.gz" --exclude='*' \
    "stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/" \
    ${partner}-${SHORT_VERSION}-rc${RC}
  rsync -av -e ssh --include="*.dmg" --include="*.exe" --include="*.tar.gz" --exclude='*' \
    stage.mozilla.org:firefox-${VERSION}/batch-cjk-repacks/original_builds/${partner}/ \
    ${partner}-${SHORT_VERSION}-rc${RC}
  ./verify_l10n.sh ${partner}-${SHORT_VERSION}-rc${RC}
  rsync -av -e ssh --include="*.en-US.mac.dmg" --include="*en-US.win32.installer.exe" --include="*.en-US.linux-i686.tar.gz" --exclude='*' \
    stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ \
    ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}
  rsync -av -e ssh --include="*-${partner}.*.dmg" \
    --include="*-${partner}.*.exe" --include="*-${partner}.*.tar.gz" --exclude='*' \
    stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${PREVIOUS_VERSION}-candidates/rc${PREVIOUS_RC}/ \
    ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}
  ./verify_l10n.sh ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC}
  diff -r ${partner}-${SHORT_PREVIOUS_VERSION}-rc${PREVIOUS_RC} ${partner}-${SHORT_VERSION}-rc${RC} > l10n_metadiff_${partner}-${SHORT_VERSION}-rc${RC}.txt
done

# look for missing files or binary diffs
grep "^Only" l10n_metadiff_*-${SHORT_VERSION}-rc${RC}.txt
grep "^Binary" l10n_metadiff_*-${SHORT_VERSION}-rc${RC}.txt

Partner repack QA tests

Send the metadiff and the location of the repacked builds to QA for testing.

Updates

Prepare the update from the previous releases to this release.

Review CVS:/mofo/release/patcher/moz*-branch-patcher2.cfg

create new past-update for previous update path
set up current-update to match new update path
create a new <release> section
get buildId from http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page
verify that all locales in shipped-locales are present

Seek review for changes.

# ssh cltbld@prometheus-vm.mozilla.org
mkdir /builds/${VERSION}-updates/
cd /builds/${VERSION}-updates/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d patcher mozilla/tools/patcher
# config lives in private repo
cvs -d cltbld@cvs.mozilla.org:/mofo co -d config release/patcher/moz180-branch-patcher2.cfg
cd patcher
export CVSROOT=cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=${PRODUCT} \
  --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}_build-tools.log
# download complete MARs
./patcher2.pl --download --app=${PRODUCT} \
  --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}-download.log
# Create partial patches and snippets
./patcher2.pl --create-patches --app=${PRODUCT} \
  --config=../config/moz180-branch-patcher2.cfg 2>&1 | tee ${PRODUCT}-create-patches.log

# prepare aus2-staging
# ssh aus2-staging.mozilla.org
cd /opt/aus2/incoming/3-staging
tmpdir="`date +%Y%m%d`-${SHORT_PRODUCT}-${VERSION}"
sudo mkdir ${tmpdir}-test
sudo chown cltbld ${tmpdir}-test
sudo mkdir ${tmpdir}
sudo chown cltbld ${tmpdir}

# copy updates from prometheus-vm.mozilla.org
# ssh prometheus-vm.mozilla.org
cd /builds/${VERSION}-updates/patcher/temp/${PRODUCT}/${PREVIOUS_VERSION}-${VERSION}/
tmpdir="`date +%Y%m%d`-${SHORT_PRODUCT}-${VERSION}"
rsync -nav -e "ssh -i $HOME/.ssh/aus" aus2.test/ \
  aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}-test/
rsync -nav -e "ssh -i $HOME/.ssh/aus" aus2/ \
  aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}/

# look at the files that would have been copied, it should only be
# betatest and releasetest directories for the first rsync and
# only beta and release directories for the second rsync
rsync -av -e "ssh -i $HOME/.ssh/aus" aus2.test/ \
  aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}-test/
rsync -av -e "ssh -i $HOME/.ssh/aus" aus2/ \
  aus2-staging.mozilla.org:/opt/aus2/incoming/3-staging/${tmpdir}/

### quick verification
# ensure that there are only test channels
# ssh aus2-staging.mozilla.org
cd /opt/aus2/incoming/3-staging/${tmpdir}-test
find . -type f | grep -v test
# last command should return no output

# push partial MARs to stage
# ssh prometheus-vm.mozilla.org
# NOTE - this will also be pushed to the live server!
#
# Verify that this command only copied what you want, 
# and remove the "-n". 
rsync -n -av -e ssh ftp/ stage.mozilla.org:/home/ftp/pub/
# In particular, verify that no "complete" MARs are sent.
rsync -av -e ssh ftp/ stage.mozilla.org:/home/ftp/pub/
# errors about "failed to set times ... Operation not permitted" are ok

# push snippets to aus2-staging 
# ssh to aus2-staging.mozilla.org
# NOTE - this will also be pushed to the live server!
#
# backup current settings
rsync -av ../../3/ ~/3-backup_${tmpdir}-test/

# Verify that this command only copied what you want, 
# and remove the "-n":
rsync -n -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/
# In particular, verify that no non-test channels are present.
rsync -n -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/ | grep -v test
# Push to production
sudo rsync -av ${AUS_PRODUCT}/ ../../3/${AUS_PRODUCT}/

Update verification

Verify that all updates are being served up on the expected channels.

Run the following on bm-xserve01:

# convert version number
SHORT_VERSION=`echo ${VERSION} | sed 's/\.//g'`
SHORT_PREVIOUS_VERSION=`echo ${PREVIOUS_VERSION} | sed 's/\.//g'`

mkdir -p /builds/${PRODUCT}-${VERSION}-verification/updates
cd /builds/${PRODUCT}-${VERSION}-verification/updates
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d updates mozilla/testing/release/updates/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/

Customize updates/updates.cfg to contain the channels you are interested in testing. It should only have lines that are "channel=releasetest". It should have a line for each platform/version combination, but no lines for the current release. The lines for the previous release should have "from=" and "to=" parameters. Old versions should not. Don't forget to update the build_id parameter.

cd /builds/${PRODUCT}-${VERSION}-verification/updates/updates
./verify.sh -t > ${PRODUCT}-${VERSION}-rc${RC}.log
grep ^HTTP/1.1 ${PRODUCT}-${VERSION}-rc${RC}.log | grep -v -E "200 OK|304 Not Modified|302 Found"
# The last command should have no output unless there were errors

Send email to QA letting them know that the updates are ready to test on the betatest and releasetest channels.

Stage release

Prepare staging area

Prepare the staging directory for the release.

NOTE: This has changed- you need to use the new GPG key attached to the bug. joduinn 29aug2007

# Create the staging directory.
cd ~/
mkdir -p ${PRODUCT}-${VERSION}/; cd ${PRODUCT}-${VERSION}/
mkdir -p batch-skel/stage/

# Create the contrib and contrib-localized directories with expected access 
# rights.
cd ~/${PRODUCT}-${VERSION}/batch-skel/stage/
mkdir -p contrib contrib-localized
chown cltbld:${PRODUCT} contrib*
chmod 2775 contrib*

# NOTE - should have a standard "master" copy somewhere else
# Copy the KEY file from the previous release directory.
# rsync -a /home/ftp/pub/${PRODUCT}/releases/1.5/KEY .
# 
# The new KEY is attached to bug 377781.
# or can be copied from 
rsync -av  /home/ftp/pub/firefox/releases/2.0.0.9/KEY  .

# Prepare the merging directory.
cd ~/${PRODUCT}-${VERSION}/
mkdir -p stage-merged/
rsync -av batch-skel/stage/ stage-merged/

Collect builds

Collect the release files onto stage.mozilla.org.

mkdir -p ~/${PRODUCT}-${VERSION}/batch1/prestage
cd ~/${PRODUCT}-${VERSION}/batch1/prestage
rsync -Lav /home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/rc${RC}/ .
find . -type f -exec chmod -v 644 {} \;
cd ..

Remove unreleased builds

Remove the unneeded files.

cd ~/${PRODUCT}-${VERSION}/batch1
rsync -av prestage/ prestage-trimmed/
cd prestage-trimmed/
rm -v *.zip *.log *.txt
rm -rv unsigned/
# is this still needed?
find . -name xforms.xpi -exec rm -fv {} \;
find . -type d -name "*-xpi" -empty -exec rmdir {} \;
# ja-JP-mac is the JA locale for mac, do not ship ja
rm -v *.ja.mac*
rm -v mac-xpi/ja.xpi
# ja is the JA locale for win32/linux, do not ship ja-JP-mac
find . -name "*ja-JP-mac.win32*" -exec rm -fv {} \;
find . -name "*ja-JP-mac.linux*" -exec rm -fv {} \;
find . -name "en-US.xpi" -exec rm -fv {} \;
rm -v linux-xpi/ja-JP-mac.xpi
rm -v windows-xpi/ja-JP-mac.xpi
cd ../

Rename builds

Rename the files. If its an Alpha/Beta, do this manually, otherwise use groom-files below.

rsync -Lav prestage-trimmed/ stage/
cd stage/
rm *.mar
~/bin/groom-files --short=${VERSION} .
mv -v linux-xpi linux-i686/xpi
mv -v windows-xpi win32/xpi
mv -v mac-xpi mac/xpi
cd ../

Verify locales

cd stage
~/bin/verify-locales.pl -m ../../batch-source/rc${RC}/mozilla/${APP}/locales/shipped-locales
cd ../

It's normal to get messages like this

Missing XPI locale for platform 'linux': en-US
Missing non-locale XPI: linux-i686/xpi/browser
Missing non-locale XPI: linux-i686/xpi/talkback
Missing non-locale XPI: linux-i686/xpi/xpcom
Missing non-locale XPI: linux-i686/xpi/adt
Missing XPI locale for platform 'win32': en-US
Missing non-locale XPI: win32/xpi/browser
Missing non-locale XPI: win32/xpi/talkback
Missing non-locale XPI: win32/xpi/xpcom
Missing non-locale XPI: win32/xpi/adt
Missing XPI locale for platform 'osx': en-US
Missing non-locale XPI: mac/xpi/browser
Missing non-locale XPI: mac/xpi/talkback
Missing non-locale XPI: mac/xpi/xpcom
Missing non-locale XPI: mac/xpi/adt

because we no longer ship an en-US.xpi or the app components as xpi files. Any other messages (eg missing or extra locales) should be fixed before continuing.

Sign builds

See here for current Fx2.0, Tb2.0, Fx3.0 signing instructions.

Define the variables

In order to generalise this document, we'll use environment vars for substitutions. For example:

PRODUCT=firefox
VERSION=2.0.0.15
BUILD=1
TAG=FIREFOX_2_0_0_15_RELEASE

You should adjust these for the release you are working on, and enter them at the Cygwin prompt.

Download files to keymaster

After logging into keymaster:

check there is enough free disk space
then run the following commands *on* keymaster to download the files to keymaster.

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
rsync -e ssh -av cltbld@stage.mozilla.org:/data/cltbld/${PRODUCT}-${VERSION}/batch1/stage-signed/ ./stage-unsigned/
rsync -av stage-unsigned/ stage-signed/

Sign win32 EXEs

first, start signcodepwd.exe (in seperate window)

 cd ~/signing-work/${PRODUCT}-${VERSION}/stage-signed/win32
 sign-release .
 verify-signatures .

 cd ../

Sign all files with PGP key

Sign all of the release files with the PGP key into detached signatures.

cd ~/signing-work/${PRODUCT}-${VERSION}/stage-signed
sign-files .

Verification

Verify that there is a detached signature for each file.

find -not -name "*.xpi" -not -name "*.asc" -type f | wc -l
find -name "*.asc" -type f | wc -l

Upload the signatures back to stage-signed/

Verify that only .exe and .asc files are modified.

cd ~/signing-work/${PRODUCT}-${VERSION}
rsync -n -e ssh -av stage-signed/ cltbld@stage.mozilla.org:${PRODUCT}-${VERSION}/batch1/stage-signed/ | grep -v asc | grep -v exe
rsync -e ssh -av stage-signed/ cltbld@stage.mozilla.org:${PRODUCT}-${VERSION}/batch1/stage-signed/

Merge files

Merge update (MAR) files

The following steps are to publish the update files for the release.

cd ~/${PRODUCT}-${VERSION}/
mkdir -p batch1/mar/stage
cd batch1/mar/stage
rsync -av ../../prestage-trimmed/ ./
find -type f -not -regex '.*.mar$' -exec rm -fv {} \;
find -type d -empty -exec rmdir -v {} \;
~/bin/groom-files --short=${VERSION} .

rsync -n -av update /data/cltbld/${PRODUCT}-${VERSION}/stage-merged/

Verify that only the files you expect are copied.

 rsync -av update /data/cltbld/${PRODUCT}-${VERSION}/stage-merged/

Merge signed files

Merge the signed file set into stage-merged.

cd ~/${PRODUCT}-${VERSION}/
rsync -n -av batch1/stage-signed/ stage-merged/

Verify that only the files you expect are copied.

rsync -av batch1/stage-signed/ stage-merged/
cd stage-merged/
~/bin/checksum-files .

Update bouncer links

Create bouncer (http://download.mozilla.org/admin) links for all products (MARs and builds).

Push the button

NOTE - copying to FTP must be done 12 hours in advance of the release announcement to ensure that mirrors are synchronized.

Make the release live

Make the release live.

Create the FTP directory.

mkdir -p /home/ftp/pub/${PRODUCT}/releases/${VERSION}

Copy the staging directory's files into the FTP directory.

cd ${PRODUCT}-${VERSION}/
rsync -av -n stage-merged/ /home/ftp/pub/${PRODUCT}/releases/${VERSION}/
# Verify that the output is what you expect, e.g. the expected directories and files are copied to the expected location.

rsync -av stage-merged/ /home/ftp/pub/${PRODUCT}/releases/${VERSION}/

Move the "latest" symlink

cd /home/ftp/pub/${PRODUCT}/releases
rm latest && ln -s ${VERSION} latest

Update the mozilla-current rsync module, which only has the current releases. On stage.m.o, modify /etc/rsyncd-mozilla-current.exclude to add the new release, replacing the previous one. IT & Nick can do this.

Request QA sign-off that build IDs in the release files match expectations.

Make the update live

Make the update live.

https://intranet.mozilla.org/Build:Automatic_Updates_for_a_Release

ssh aus2-staging.mozilla.org
cd /builds/${VERSION}-updates/3-staging/

### quick verification
# ensure that test channels have the same content as release channels
# TODO
# compare production differences
diff -r ${AUS_PRODUCT}-${tmpdir}/${AUS_PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/

# ensure that only the files you expect are copied!
rsync -n -av ${AUS_PRODUCT}-${tmpdir}/${PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/

# push to production
rsync -av ${AUS_PRODUCT}-${tmpdir}/${AUS_PRODUCT}/ ../3/incoming/${AUS_PRODUCT}/

Reset for next release

Bump version number

Bump mozilla version number to ${NEXT_VERSION}pre/${PLATFORM_NEXT_VERSION}pre.

Three files need to be updated to the right version number in cvsroot/mozilla/ :

${APP}/app/module.ver
${APP}/config/version.txt
config/milestone.txt

File a "version bump" bug and attach a patch before committing.

ReleaseEngineering/Unified Release Process

Contents

Release prep

Create sign-off sheet

Configure each build

Bump version number

Tag the release

Set up the system to tag

Tagging cvsroot/mozilla

Tagging mofo/talkback/fullsoft

Tagging l10n/l10n

Update repository information

Build en-US

Build

Verify build logs

Copy en-US builds to candidates directory

Archive the source

Build l10n

Configure each build

Build

Verify build logs

Minimum locales verification

Gather l10n files

l10n verification

l10n QA tests

Signing windows files

Repackaging

Verify partner repacks

Partner repack QA tests

Updates

Update verification

Stage release

Prepare staging area

Collect builds

Remove unreleased builds

Rename builds

Verify locales

Sign builds

Define the variables

Download files to keymaster

Sign win32 EXEs

Sign all files with PGP key

Verification

Upload the signatures back to stage-signed/

Merge files

Merge update (MAR) files

Merge signed files

Update bouncer links

Push the button

Make the release live

Make the update live

Reset for next release

Bump version number

Navigation menu

Search