From MozillaWiki
Jump to: navigation, search

Using Bootstrap (RELEASE_AUTOMATION_M7_1) but starting steps manually.

Build Engineers


Version/config bump bug

bug 417070

Bonsai queries


Using the GECKO181_20080128_RELBRANCH.

Module Tag Pull date
cvsroot/mozilla THUNDERBIRD_2_0_0_12_RELEASE GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST
THUNDERBIRD_2_0_0_9_RC1 GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST
l10n/l10n THUNDERBIRD_2_0_0_12_RELEASE GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST
THUNDERBIRD_2_0_0_12_RC1 GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST

NB: Talkback not listed, we're using pre-build binaries (bug 403701). It was still tagged because this change is very new, and the automation needs remedial lessons.

Build data

See page history for BuildIDs of prior RCs

Type Build ID SHA1 Push date Build machine
[Windows installer] RC1:2008021305 patrocles (cerberus-vm for l10n)
[Mac compressed] RC1:2008021304 xserve02
[Linux compressed] RC1:2008021305 crazyhorse (karma for l10n)

Detailed Notes


Bootstrap tag: RELEASE_AUTOMATION_M7_1

Setup before starting:

  • used existing version bump bug to attached diff for tb-moz18-bootstrap.cfg
  • ensure that machines have enough resources (bug 393274)
    • On production-1.8-master, remove private and public areas:
      • rm -rf /data/cltbld/firefox-*
      • rm -rf /builds/tags/* - did not do this, plenty of space
      • in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
      • pull in files with rsync -av /data/ftp/pub/thunderbird/nightly/
    • On the build machines (see table aboue), checked that there was sufficient working space (~ 1GB), and stopped tinderbox. Also verified there is space available on bm-xserve04 for l10n-verify.
    • On prometheus-vm, checked that there was space for another update run (4GB)
  • on each machine, setup release automation using the tag above
cd /builds/release   # or /cygdrive/X/builds/release
cvs -nq up -r RELEASE_AUTOMATION_M7_1
# check for merges (M) or conflicts (C); if any found, check for local changes
cvs -q up -r RELEASE_AUTOMATION_M7_1
cd configs
cvs up -A     # in case we need to make changes later
cd ..
rm bootstrap.cfg logs
ln -s configs/tb-moz18-bootstrap.cfg bootstrap.cfg
mkdir -p old-logs/logs.tb20012/rc1
ln -s old-logs/logs.tb20012/rc1 logs

perl -c release

Some machines don't have the perl module Config::General so comment out

use Config::General;

at the top of Bootstrap/Step/ Only need this on prometheus-vm anyway, which has it.


On production-1.8-master in /builds/release:

./release -o Tag 2>&1 | tee logs/release-Tag-RC1.log

Had to come back to fix the 45 files in l10n/uk/editor, which had been tagged with MOZILLA_1_8_BRANCH by a localiser, rather than properly branched. The were all at rev 1.1 and had seen no changes on trunk, so it was straightforward to delete the existing tag, branch to MOZILLA_1_8_BRANCH, update to that, branch to the relbranch, and tag ..._RELEASE and ..._RC1.


On production-1.8-master in /builds/release:

./release -o Source 2>&1 | tee logs/release-Source-RC1.log


On each tinderbox, cd to /builds/release (prefix with /cygdrive/e/ on Windows) and

./release -o TinderConfig 2>&1 | tee logs/release-TinderConfig.log
./release -o Build 2>&1 | tee logs/release-Build.log

Needed to comment out lines 43-49 of on Windows t'box, since Cygwin complains "Permissied denied" trying to execute those lines. The disks are already mounted in binmode.


On the tinderboxes in the table above,

./release -o Repack -e 2>&1 | tee logs/release-Repack-Execute.log
./release -o Repack -p 2>&1 | tee logs/release-Repack-Push.log
./release -o Repack -a 2>&1 | tee logs/release-Repack-Announce.log

Had to rerun Linux & Mac for the uk tagging issue (see above), but was able to manually checkout the files on the win32 box before it got to that locale.

After win32 l10n, had to set cerberus-vm back to text mode

mount -t -sc /cygdrive

so that tinderbox would run. This box was very slow, taking 7 hours to complete the repack.


As per the doc.


Ran this on bm-xserve05,

./release -o Repack -v 2>&1 | tee logs/release-Repack-Verify.log

Update Generation

Generated on production-prometheus-vm,

./release -o PatcherConfig 2>&1 | tee logs/release-PatcherConfig.log
./release -o Updates -e 2>&1 | tee logs/release-Updates-Execute.log
./release -o Updates -p 2>&1 | tee logs/release-Updates-Push.log

Push dirs were 20080218-Thunderbird- (& -test & -beta)

NB: DisableCompleteJump will need to be manually added to the rc block of the patcher config if we do an rc2, then regenerate the snippets.

QA found a problem with the details url - bug 418450. So we fix the config and regenerate the snippets:

cd /builds/updates/thunderbird-
cvs up
cd ../patcher/temp/thunderbird/
mkdir wrong-details-url
mv aus2* wrong-details-url/
cd ../../..
./ --create-patchinfo --app=thunderbird \
 --config=../config/moz18-branch-patcher2.cfg 2>&1 | tee thunderbird-create-patchinfo.log

Do some quick verification

cd temp/thunderbird/
diff -r aus2.test/ wrong-details-url/aus2.test/ | less
diff -r aus2.beta/ wrong-details-url/aus2.beta/ | less
diff -r aus2/ wrong-details-url/aus2/ | less

Only the detailsURL differs.

Push to aus2-staging as 20080219-Thunderbird-{-test,-beta,}, then manually push 20080219-Thunderbird- live.

Update Verify

Run on the tinderboxes

./release -o Updates -v 2>&1 | tee logs/release-Updates-Verify.log

On crazyhorse, verify failed because wget is too old. Removed --no-check-certificate argument in updates/, common/, and common/, then reran.

patrocles doesn't have wget, so used production-pacifica-vm instead - needed to modify the win32_logDir from /cygdrive/e/... to /cygdrive/c/... in the bootstrap config.

Get expected FAIL's for 2.0b2/gu-IN/all platforms and 2.0b2/he/mac - no longer shipped. Also non-binary warnings for installed-chrome.txt and removed-files, which we no longer ignore.

Updates to Beta Channel

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ screen
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ ~/bin/pushsnip 20080219-Thunderbird-


On production-1.8-master

./release -o Stage 2>&1 | tee logs/release-Stage.log

Backed up /data/cltbld/bin/ before running this, and modified the three regex's on the file name (firefox -> thunderbird, Firefox -> Thunderbird). This is bug 409232.

Sign Installers

  • NOTE - production-1.8-master is not used from here on
  • pull stage:/data/cltbld/thunderbird- to signing machine
  • follow installer-signing-instructions here
  • push signed bits back to same location on stage
  • complete stage-merged:
# on stage
cd /data/cltbld/thunderbird-
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/thunderbird-
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:thunderbird .
chmod 644 *SUMS


  • get the last formal "go" from QA/Dev/website/IT/release-drivers
  • push the stage-merged directory to the releases area (had to create target dir as root first):
# on stage
rsync -av /data/cltbld/thunderbird- /home/ftp/pub/thunderbird/releases/
  • After pushing live, move forward the "latest" and "latest-2.0" symlinks.
 cd /home/ftp/pub/thunderbird/releases/
 rm latest
 ln -s latest
 rm latest-2.0
 ln -s latest-2.0
  • Update stage:/etc/rsyncd-mozilla-current.exclude - not critical that this happens immediately.

Update Bouncer

  • Add entries to

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-thunderbird-*.cfg | sed 's/betatest/releasetest/' > update.cfg
./ -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080219-Thunderbird-
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080219-Thunderbird-$a");'

Enable update channel

  • Wait for QA to finish testing update snippets on releasetest channel, then enable update snippets on release channel
 # on aus2-staging
 # put snippets on release/live channel
 $ screen
 $ sudo su - cltbld
 $ cd /opt/aus2/snippets/staging
 $ ~/bin/pushsnip 20080219-Thunderbird-