Build:Release Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(prep for 2.0.0.12 release)
No edit summary
 
(9 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Obsolete RelEng Page|It was last edited in 2008, and is thus certainly out of date by now.}}
=Intro=
=Intro=
Firefox and Thunderbird releases are currently done using the [http://wiki.mozilla.org/Build:Release_Automation#Bootstrap Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.
Firefox and Thunderbird releases are currently done using the [http://wiki.mozilla.org/Build:Release_Automation#Bootstrap Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.
Line 20: Line 22:
=Current status=
=Current status=


For the Firefox 2.0.0.12, we are using Buildbot to drive the release. Instead of a human operater logging into [http://wiki.mozilla.org/index.php?title=Build:Release_Automation#Roles_and_resource_requirements each machine] used in the release process, the machines run Buildbot slaves instead. Most of the slaves simply check out and run Bootstrap, at this point.
This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the [[Releases]] page.
 
Both the [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/staging/ staging] and [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/production production] configs are checked into CVS.
 
There are still several manual processes, which we are working on automating:
 
* Buildbot config
** if necessary, tag new version of mozilla/tools/release (used RELEASE_AUTOMATION_M7)
*** make sure buildbot-configs/automation/production/master.cfg uses this tag
** need to file bug (e.g. {{Bug|414327}}) and attach diff for bootstrap.cfg (e.g. [http://mxr.mozilla.org/mozilla/source/tools/release/configs/fx-moz18-bootstrap.cfg fx-moz18-bootstrap.cfg])
** need to "cvs update" /home/buildmaster/Automation/buildbot-configs/ and /home/buildmaster/Automation/bootstrap-configs/ after checkin
** need to manually insert passwords into master.cfg, as they are intentionally not in the checked-in version.
** make sure Tinderbox is up-to-date on all slaves {{Bug|397554}}
*** used RELEASE_AUTOMATION_M7 (after tagging mozilla/tools/tinderbox with this)
** reconfig buildbot master (<tt>buildbot reconfig `pwd`</tt>), so new automation tag is detected
** ensure that machines have [http://wiki.mozilla.org/Build:Release_Automation#Roles_and_resource_requirements enough resources] {{Bug|393274}}
** cd /data/cltbld/bin and run "cvs up" to update mofo scripts
*** FIXME - these should really be auto-updated and moved to public repo
** kick off buildbot (run as cltbld):
*** buildbot sendchange --master=localhost:9989 -u joduinn -m"Firefox 2.0.0.12 RC1" release


* Tag
All configs are checked into CVS under [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/ mozilla/tools/buildbot-configs/automation].
** *only applicable for relbranch-based builds* - Bootstrap::Tag::Bump does not handle already-bumped files on relbranch
*** need to make client.mk *_CO_TAGs use MOZILLA_1_8_BRANCH
*** need to add "2.0.0.12pre" tags to version bump files:
**** browser/app/module.ver browser/config/version.txt config/milestone.txt
 
* Sign
** still manual
** need to sync signed bits manually back to build-console e.g. as cltbld@build-console:
*** rsync -av stage.mozilla.org:/home/ftp/pub/firefox/nightly/2.0.0.12-candidates/ /home/ftp/pub/firefox/nightly/2.0.0.12-candidates/
 
* Updates
 
* Stage
** created bouncer links manually {{Bug|414327}}
** created "latest" and "latest-2.0" symlinks manually after final release


=Roles and resource requirements=
=Roles and resource requirements=
Line 102: Line 70:
  make cvsmirror
  make cvsmirror


* Update bootstrap and buildbot configs. These are symlinked from bootstrap-configs and buildbot-configs checkouts (of mozilla/tools/release/configs/ and mozilla/tools/buildbot-configs/automation/staging/, respectively).
* Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).


As buildmaster@staging-build-console:
As buildmaster@staging-build-console:
Line 108: Line 76:
  cd /home/buildmaster/TestBot
  cd /home/buildmaster/TestBot
  buildbot stop `pwd`
  buildbot stop `pwd`
cd bootstrap-configs && cvs up && cd ../
  cd buildbot-configs && cvs up && cd ../
  cd buildbot-configs && cvs up && cd ../
  buildbot start `pwd`
  buildbot start `pwd`
Line 116: Line 83:
  cd /data/cltbld/bin
  cd /data/cltbld/bin
  cvs up
  cvs up
* Update the verify configs, removing the latest entry as appropriate.
These configs will be bumped during the update verify runs. e.g. for 1.8 branch Firefox:
cvs -d /builds/cvsmirror.clean/cvsroot co mozilla/testing/release/updates
cd mozilla/testing/release/updates
# remove top two lines, from moz18-firefox-*.cfg e.g.:
# 2.0.0.10 linux
release="2.0.0.10" product="Firefox" platform="Linux_x86-gcc3" build_id="2007111504" locales="af ar be bg ca cs da de el en-GB en-US es-AR es-ES eu fi fr fy-NL ga-IE gu-IN he hu it ja ka ko ku lt mk mn nb-NO nl nn-NO pa-IN pl pt-BR pt-PT ro ru sk sl sv-SE tr uk zh-CN zh-TW" channel="betatest" from="/firefox/releases/2.0.0.10/linux-i686/%locale%/firefox-2.0.0.10.tar.gz" aus_server="https://aus2.mozilla.org" ftp_server="stage.mozilla.org/pub/mozilla.org" to="/firefox/nightly/2.0.0.11-candidates/rc1/firefox-2.0.0.11.%locale%.linux-i686.tar.gz"
cvs ci -m"remove latest release from staging config"


* Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
* Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
Line 134: Line 92:


For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.
For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.
* MOZILLA_1_8_BRANCH ONLY
** The Talkback symbol server is hardcoded in /builds/cvsmirror.clean/mofo/talkback/fullsoft/Makefile.in, replace all occurrences of "talkback-upload.mozilla.org" with "staging-build-console.build.mozilla.org".


* re-enable cltbld's nightly cronjob
* re-enable cltbld's nightly cronjob
=Enhancements=
* ([https://bugzilla.mozilla.org/show_bug.cgi?id=394507  bug 394507]) should set buildbot up to mail based on any failures, currently just depend on bootstrap
* ({{bug|372746}}) Automatically configure bouncer
* ([https://bugzilla.mozilla.org/show_bug.cgi?id=373995 bug 373995]) l10n needs the URL it downloads builds from to be configurable as well
* ([https://bugzilla.mozilla.org/show_bug.cgi?id=394498 bug 394498]) should report on mirror saturation after release
* ({{bug|397554}}) Automatically check out, set up, and keep Tinderbox installs up to date
* [http://buildbot.net/trac/ticket/68 buildbot bug#68] buildbot default timeout too short. 5sec isnt always enough, and you can get a "timed out" message in the slave logs, even though slave started "normally".
* [http://buildbot.net/trac/ticket/85 buildbot bug#85] sometimes buildmaster sees buildslave correctly, confirms ping ok, but never assigns pending work to the slave. Doing "buildmaster refresh" is not enough, you need to do "buildmaster stop/start". Restarting the slave does not help.
* [http://buildbot.net/trac/ticket/92 buildbot bug#92] on win32, console output is not logged (goes to the DOS console running buildbot :( )
* [http://buildbot.net/trac/ticket/77 buildbot bug#77] file buildbot bug to handle kill on win32. Add details linking to bsmedberg fix.
* [http://buildbot.net/trac/ticket/67 buildbot bug#67] link to history for old builds at bottom of page (ala tinderbox server).
* [http://buildbot.net/trac/ticket/69 buildbot bug#69] meta-refresh tag for waterfall page
* [http://buildbot.net/trac/ticket/78 buildbot bug#78] buildbot UI to contain way to force build dependent steps instead of just doing current step.
* [http://buildbot.net/trac/ticket/91 buildbot bug#91] When using the CVS Source step on a Mac OSX slave, if a CVS directory is found on the path, buildbot will attempt to use it as if it were a CVS binary.
* [http://buildbot.net/trac/ticket/88 buildbot bug#88] steps which start within a few seconds of each other show as same start time on waterfall page
* (needs bug filed) tinderbox symbol server should be configurable


=Setup notes (historical)=
=Setup notes (historical)=

Latest revision as of 20:39, 17 March 2017

Warning: This RelEng page is obsolete!
It was last edited in 2008, and is thus certainly out of date by now.

Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Buildbot calls bootstrap, parallelizing and serializing where needed.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Buildbot

Buildbot is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.

We have a vendor branch in mozilla/tools/buildbot, based on Buildbot's 0.7.5 release.

Mozilla-specific Buildbot install instructions

Buildbot user manual

Current status

This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the Releases page.

All configs are checked into CVS under mozilla/tools/buildbot-configs/automation.

Roles and resource requirements

  • buildbot master
    • keeps logs, manages overall process
  • ftp/stage.m.o
    • fileserver, both public and private areas
    • FTP candidates - 20GB storage
    • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
    • FTP private staging - 20GB storage
      • e.g. stage:firefox-2.0.0.4/
    • FTP release - 6GB storage
      • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
  • "tagging" builder
    • checks out source and applies tag
    • 2GB storage
      • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
  • "source archive" builder
    • builds source archive and pushes for QA
  • "linux/mac/win32 firefox builders"
    • builds firefox and pushes for QA
    • l10n/update verification
    • needs 2GB memory, 10GB storage (each)
      • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
  • "updates builder"
    • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
    • updates - 1GB memory, 5GB storage
    • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
  • "stage builder"
    • creates private staging area on FTP, renames files for release
    • see "fileserver" requirements, above
  • Automatic Update Server (AUS), aus2.m.o
    • 10GB for config files, backups and staging area
    • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Updating Staging release version

  • Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. bug 407672
  • As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.
  • As cltbld@staging-build-console, refresh cvsmirror by doing:
cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror
  • Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).

As buildmaster@staging-build-console: 

# pull configs and restart buildbot
cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`

As cltbld@staging-build-console: 

# update mofo scripts
cd /data/cltbld/bin
cvs up
  • Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux

For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.

  • re-enable cltbld's nightly cronjob

Setup notes (historical)

Retrieved from "https://wiki.mozilla.org/index.php?title=Build:Release_Automation&oldid=1166043"