Build:Release Automation

Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Buildbot calls bootstrap, parallelizing and serializing where needed.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Buildbot

Buildbot is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.

We have a vendor branch in mozilla/tools/buildbot, based on Buildbot's 0.7.5 release.

Mozilla-specific Buildbot install instructions

Buildbot user manual

Current status

For the Firefox 2.0.0.8 release, we are using Buildbot to drive the release. Instead of a human operater logging into each machine used in the release process, the machines run Buildbot slaves instead. Most of the slaves simply check out and run Bootstrap, at this point.

Both the staging and production configs are checked into CVS.

There are still several manual processes, which we are working on:

• Buildbot config
  • if necessary, tag new version of mozilla/tools/release (used RELEASE_AUTOMATION_M5_3)
    • make sure buildbot-configs/automation/production/master.cfg uses this tag
  • need to file bug (e.g. bug 393264) and attach diff for bootstrap.cfg (e.g. fx-moz18-bootstrap.cfg)
  • need to "cvs update" /home/buildmaster/Automation/buildbot-configs/ and /home/buildmaster/Automation/bootstrap-configs/ after checkin
  • need to manually insert passwords into master.cfg, as they are intentionally not in the checked-in version.
  • make sure Tinderbox is up-to-date on all slaves bug 397554
    • used RELEASE_AUTOMATION_M5_3
  • ensure that machines have enough resources bug 393274
  • kick off buildbot (run as cltbld):
    • buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.8 RC1" release
  • it's not possible to do dependent schedulers with a forced tag bug 394963
  • need to disable updateverify depscheduler until auto-config for update verify is done bug 373995, and Update step is able to auto-deploy.

• Tag -
  • had to manually tag based on GECKO181_20070712_RELBRANCH bug 396290
    • NOTE - RC1/RC2 respin case fixed/tested; RC1 firedrill should work, not yet tested

• Source
  • permissions for generated source tarball are incorrect, (0700 should be 0644). For FF2.0.0.8 release, automation created the source tarball with the correct 0644. bug 399628
  • must be run on stage, need to rewrite source step bug 394034
  • manually sync build-console and stage bug 396438

• Build
  • manually sync build-console and stage bug 396438

• Repack
  • had to fall back to cerberus-vm due to EOL problems bug 397842
  • manually sync build-console and stage bug 396438
  • "scp -r" does not work on pacifica-vm, need to upgrade or use something else (e.g. rsync), fixed in tinderbox

• Sign
  • still manual
  • need to sync signed bits manually back to build-console e.g. as cltbld@build-console:
    • rsync -av stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.8-candidates/ /home/ftp/pub/firefox/nightly/2.0.0.8-candidates/

• Updates
  • call push, "./release -o Updates -p", manually (should be fixed on bootstrap trunk bug 389206)
  • had to correct permissions for both snippets and MARs (should be fixed on bootstrap trunk bug 399628)
  • update verification config is still manual (should be fixed on bootstrap trunk bug 373995)
  • had to change stagingServer to "stage" and re-run configs bug 396438
  • manually sync build-console and stage bug 396438

• Stage
  • had to correct permissions (should be fixed on bootstrap trunk bug 399628)
  • need to rsync /data/cltbld/firefox-2.0.0.8/ stage:/data/cltbld/firefox-2.0.0.8/ (not covered by bug 396438).
  • created bouncer links manually bug 372746
  • created "latest" and "latest-2.0" symlinks manually after final release

Roles and resource requirements

• buildbot master
  • keeps logs, manages overall process
• ftp/stage.m.o
  • fileserver, both public and private areas
  • FTP candidates - 20GB storage
  • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
  • FTP private staging - 20GB storage
    • e.g. stage:firefox-2.0.0.4/
  • FTP release - 6GB storage
    • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
• "tagging" builder
  • checks out source and applies tag
  • 2GB storage
    • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
• "source archive" builder
  • builds source archive and pushes for QA
• "linux/mac/win32 firefox builders"
  • builds firefox and pushes for QA
  • l10n/update verification
  • needs 2GB memory, 10GB storage (each)
    • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
• "updates builder"
  • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
  • updates - 1GB memory, 5GB storage
  • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
• "stage builder"
  • creates private staging area on FTP, renames files for release
  • see "fileserver" requirements, above
• Automatic Update Server (AUS), aus2.m.o
  • 10GB for config files, backups and staging area
  • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Updating Staging release version

• Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. bug 397425

• Disable cltbld's nightly cronjob

• Refresh cvsmirror

As cltbld@staging-build-console:

cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror

• Update bootstrap and buildbot configs. These are symlinked from bootstrap-configs and buildbot-configs checkouts (of mozilla/tools/release/configs/ and mozilla/tools/buildbot-configs/automation/staging/, respectively).

As buildmaster@staging-build-console:

cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd bootstrap-configs && cvs up && cd ../
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`

NOTE - the Talkback symbol server is hardcoded in /builds/cvsmirror.clean/mofo/talkback/fullsoft/Makefile.in, this should be changed like so:

FC_TUNNEL       = ssh -$(FC_SSH_VERSION) -f -L 8080:hal:80 $(LSSH_USER)staging-build-console.build.mozilla.org sleep 20
SYM_TUNNEL      = ssh -$(SYM_SSH_VERSION) -f -L 2222:localhost:22 $(LSSH_USER)staging-build-console.build.mozilla.org sleep 20

• re-enable cltbld's nightly cronjob

Enhancements

• (bug 394507) should set buildbot up to mail based on any failures, currently just depend on bootstrap
• (bug 372746) Automatically configure bouncer
• (bug 373995) l10n needs the URL it downloads builds from to be configurable as well
• (bug 394498) should report on mirror saturation after release
• (bug 397554) Automatically check out, set up, and keep Tinderbox installs up to date
• buildbot bug#68 buildbot default timeout too short. 5sec isnt always enough, and you can get a "timed out" message in the slave logs, even though slave started "normally".
• buildbot bug#85 sometimes buildmaster sees buildslave correctly, confirms ping ok, but never assigns pending work to the slave. Doing "buildmaster refresh" is not enough, you need to do "buildmaster stop/start". Restarting the slave does not help.
• buildbot bug#92 on win32, console output is not logged (goes to the DOS console running buildbot :( )
• buildbot bug#77 file buildbot bug to handle kill on win32. Add details linking to bsmedberg fix.
• buildbot bug#67 link to history for old builds at bottom of page (ala tinderbox server).
• buildbot bug#69 meta-refresh tag for waterfall page
• buildbot bug#78 buildbot UI to contain way to force build dependent steps instead of just doing current step.
• buildbot bug#91 When using the CVS Source step on a Mac OSX slave, if a CVS directory is found on the path, buildbot will attempt to use it as if it were a CVS binary.
• buildbot bug#88 steps which start within a few seconds of each other show as same start time on waterfall page
• (needs bug filed) tinderbox symbol server should be configurable

Setup notes (historical)

• Notes on Staging Setup
• Notes on Production Setup