Build:Release Automation

Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Currently, a human operator must:

1. log into the appropriate machine
2. check out bootstrap
3. run the appropriate bootstrap command

This must be done ~7 times (once per machine), in the right order, to produce a successful release.

Work is ongoing to enable Buildbot to drive the release automation, to enable an "end-to-end" run without human involvement to move the process from step-to-step and machine-to-machine.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Bootstrap is invoked using the "release" command, and supports a set of high-level "steps":

Tag - tag, branch, apply version bumps to all relevant files.
TinderConfig - generate tinderbox config files (mozconfig/tinder-config.pl)
Build - invoke Tinderbox client to create and en-US build and publish to FTP
Source - create a source tarball and push it to FTP
Repack - invoke Tinderbox client to create localized versions of en-US build and publish to FTP
PatcherConfig - create a Patcher config file for generating updates
Updates - invoke Patcher to create partial updates and AUS configuration
Stage - create a staging area and rename files for release
Sign - not implemented

Bootstrap Steps

A Bootstrap "step" must implement 2 required methods:

Execute - carry out the actual function of the step, e.g. Build
Verify - run an automated test

Additionally, there are 2 optional methods:

Push - upload the appropriate changes for testing, e.g. upload build to FTP
Announce - send an email announcing that the step has finished.

Using Bootstrap

If the "release" command is invoked with no parameters, it will attempt to start at the first step and call the methods in this order:

1. Execute
2. Verify
3. Push
4. Announce

As each step completes successfully, the next will be invoked.

There are several command-line options, shown by calling "release -h":

Usage: release [-l] [-s Step] [-o Step] [-e | -v | -p | -a] [-h]
    -l list all Steps
    -s start at Step
    -o only run one Step
    -e only run Execute
    -v only run Verify
    -p only run Push
    -a only run Announce
    -h this usage message

For example, to only run the Push method on the Build step:

./release -o Build -p

Roles and resource requirements

• buildbot master
  • keeps logs, manages overall process
• ftp/stage.m.o
  • fileserver, both public and private areas
  • FTP candidates - 20GB storage
  • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
  • FTP private staging - 20GB storage
    • e.g. stage:firefox-2.0.0.4/
  • FTP release - 6GB storage
    • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
• "tagging" builder
  • checks out source and applies tag
  • 2GB storage
    • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
• "source archive" builder
  • builds source archive and pushes for QA
• "linux/mac/win32 firefox builders"
  • builds firefox and pushes for QA
  • needs 2GB memory, 6GB storage (each)
    • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
• "updates builder"
  • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
  • updates - 1GB memory, 5GB storage
  • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
• "stage builder"
  • creates private staging area on FTP, renames files for release
  • see "fileserver" requirements, above
• Automatic Update Server (AUS), aus2.m.o
  • 10GB for config files, backups and staging area
  • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Notes on staging setup

Bootstrap uses a local CVS mirror, and the "tag", "source", "updates", and "stage" builders are run by a local buildslave.

The bootstrap Makefile has the following targets:

• stage/clean_stage
  • create/remove basic fileserver/tag/source/updates/stage environment
• cvsmirror/clean_cvsmirror
  • create/remove cvsmirror in /builds/cvsmirror

These targets are hard-coded to prepare for a 2.0.0.4 release.

Production setup HOWTO for linux/mac/win32

• (Win32/Mac only) install Config::General
  • FIXME - this should not be necessary, it's not actually used on these platforms

 cd /tools/dist
 wget http://search.cpan.org/CPAN/authors/id/T/TL/TLINDEN/Config-General-2.33.tar.gz 
 tar xfvz Config-General-2.33.tar.gz
 cd Config-General-2.33
 perl Makefile.PL

its ok to ignore the warning from "perl Makefile.PL": Warning: the following files are missing in your kit: t/test.rc.out

 sudo make install

• (Linux only) prepend custom GCC to the path in ~/.bash_profile

export PATH="/usr/gcc-3.3.2rh/bin:/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:$PYTHONHOME/bin:$PATH"

• create logs dir

$ mkdir -p /tools/dist/logs
$ mkdir -p /builds/logs

• look for Tinderbox directory

#linux: if tinderbox name is not "Fx-Mozilla1.8-Release" exactly, symlink it 
ln -s /builds/tinderbox/Fx-Mozilla1.8-release /builds/tinderbox/Fx-Mozilla1.8-Release

• set up Tinderbox l10n build directory

# linux
cd /builds/tinderbox/
# win32
cd /cygdrive/c/builds/tinderbox/

mkdir Fx-Mozilla-1.8-l10n-Release
cd Fx-Mozilla-1.8-l10n-Release
../mozilla/tools/tinderbox/install-links
rm build-seamonkey.pl
ln -s ../mozilla/tools/tinderbox/build-firefox.pl .
ln -s build-firefox.pl build-seamonkey.pl
rm post-mozilla.pl
ln -s post-mozilla-release.pl post-mozilla.pl

Check out tinderbox configs:

# win32
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/win32
# linux
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/linux

ln -s tinderbox-configs/mozconfig .
ln -s tinderbox-configs/tinder-config.pl . 

• Install buildbot
• running as "cltbld", install slave

#linux
$ cd ~
$ buildbot create linux-slave1 build-console.build.mozilla.org:9989 linux-slave1 password
#win32
c:\\buildtools\\python24\\scripts\\buildbot create-slave c:\\win32-slave1 build-console.build.mozilla.org:9989 win32-slave1 password

• edit the admin and host pages in ~/linux-slave1/info/

• start slave

#linux
buildbot start /home/cltbld/linux-slave1
# win32
c:\\buildtools\\python24\\scripts\\buildbot start c:\\win32-slave1

Just for testing

• Move prod ssh keys out of the way, and copy in "staging" keys:

cd ~
mv ~/.ssh ~/ssh.prod
scp cltbld@staging-prometheus-vm:~/.ssh/id_rsa .ssh/

• Move prod tinderbox-configs and put staging-build-console in Root:

# win32
cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-Release

cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot 
vi tinderbox-configs/CVS/Root

Same for l10n tinderbox build directories:

# win32
cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-l10n-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-l10n-Release

cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot  
vi tinderbox-configs/CVS/Root

Production changes

Changing roles

1. move to dedicated machines, e.g. production-prometheus-vm
2. CVS tag on linux slave or on build-console? build-console
3. l10nverify on mac slave ok, need to fix "unpack all xpis bug"
4. Mac - is identical hardware req'd? What happens if prod hardware dies? fireball still worked on, scarce PPC hardware options.

Available PPCs:

• 01 - head node
• 02 - production
• 03 - given to community
• 04 - 1.8.0
• 05 - dead
• 06 - given to community
• fireball - unknown

discussed: planned switch to Intel

Staging/Production Buildbot master differences

1. Signing - prod waits for signed bits, stage fakes w/ symlink ok
2. Bootstrap - prod pulls tag e.g. RELEASE_AUTOMATION_M5, staging pulls tip ok

Outstanding issues

1. How to handle bootstrap logs.. remove them between runs? Don't want accumulation on slaves remove at start
2. How to do mock release.. fake version (e.g. 1.2.3.4)? Early 2.0.0.7, that we know we won't release? 2007 rc1
3. "Source" and "Staging" steps - install a buildslave on stage, or stage everything on build-console? use build-console
4. Make sure QA checks e.g. top 5 extensions after Mac Intel switch

Caveats

Manual steps

NOTE - manual steps should be done in this order

1. bootstrap configuration
2. kicking off buildbot ("buildbot sendchange ...")
3. update verification config (working on this in bug 373995. For now, need to modify and check in the appropriate update configs, after all en-US builds but before updates
4. win32 signing, after win32 l10n repack but before updates
5. final installer signing