Build:Release Automation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(notes about build-console setup)
No edit summary
 
(157 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Obsolete RelEng Page|It was last edited in 2008, and is thus certainly out of date by now.}}
=Intro=
=Intro=
Firefox and Thunderbird releases are currently done using the [http://mxr.mozilla.org/seamonkey/source/tools/release/ Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.
Firefox and Thunderbird releases are currently done using the [http://wiki.mozilla.org/Build:Release_Automation#Bootstrap Bootstrap] automation scripts, which call into [http://mxr.mozilla.org/seamonkey/source/tools/tinderbox/ Tinderbox client] to do the actual build.


Currently, a human operator must:
[http://wiki.mozilla.org/Build:Release_Automation#Buildbot Buildbot] calls bootstrap, parallelizing and serializing where needed.
 
# log into the appropriate machine
# check out bootstrap
# run the appropriate bootstrap command
 
This must be done ~7 times (once per machine), in the right order, to produce a successful release.
 
Work is ongoing to enable [http://buildbot.net Buildbot] to drive the release automation, to enable an "end-to-end" run without human involvement to move the process from step-to-step and machine-to-machine.


=Bootstrap=
=Bootstrap=


[http://mxr.mozilla.org/seamonkey/source/tools/release/ Bootstrap] is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.  
[[Build:Release Automation:Bootstrap|Bootstrap]] is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.
 
Bootstrap is invoked using the "release" command, and supports a set of high-level "steps":
 
<b>Tag</b> - tag, branch, apply version bumps to all relevant files.<br>
<b>TinderConfig</b> - generate tinderbox config files (mozconfig/tinder-config.pl)<br>
<b>Build</b> - invoke Tinderbox client to create and en-US build and publish to FTP<br>
<b>Source</b> - create a source tarball and push it to FTP<br>
<b>Repack</b> - invoke Tinderbox client to create localized versions of en-US build and publish to FTP<br>
<b>PatcherConfig</b> - create a [http://mxr.mozilla.org/seamonkey/source/tools/patcher/ Patcher] config file for generating updates<br>
<b>Updates</b> - invoke Patcher to create partial updates and [http://wiki.mozilla.org/AUS AUS] configuration<br>
<b>Stage</b> - create a staging area and rename files for release<br>
<b>Sign</b> - not implemented<br>
 
==Bootstrap Steps==
 
A Bootstrap "step" must implement 2 required methods:
 
<b>Execute</b> - carry out the actual function of the step, e.g. Build<br>
<b>Verify</b> - run an automated test<br>
 
Additionally, there are 2 optional methods:
 
<b>Push</b> - upload the appropriate changes for testing, e.g. upload build to FTP<br>
<b>Announce</b> - send an email announcing that the step has finished.<br>


==Using Bootstrap==
=Buildbot=


If the "release" command is invoked with no parameters, it will attempt to start at the first step and call the methods in this order:
[http://buildbot.net Buildbot] is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.


# Execute
We have a vendor branch in [http://mxr.mozilla.org/mozilla/source/tools/buildbot/ mozilla/tools/buildbot], based on Buildbot's 0.7.5 release.
# Verify
# Push
# Announce


As each step completes successfully, the next will be invoked.
[http://wiki.mozilla.org/ReferencePlatforms/BuildBot Mozilla-specific Buildbot install instructions]


There are several command-line options, shown by calling "release -h":
[http://buildbot.net/trac/wiki/UserManual Buildbot user manual]


Usage: release [-l] [-s Step] [-o Step] [-e | -v | -p | -a] [-h]
=Current status=
    -l list all Steps
    -s start at Step
    -o only run one Step
    -e only run Execute
    -v only run Verify
    -p only run Push
    -a only run Announce
    -h this usage message


For example, to only run the Push method on the Build step:
This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the [[Releases]] page.


./release -o Build -p
All configs are checked into CVS under [http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/automation/ mozilla/tools/buildbot-configs/automation].


=Roles and resource requirements=
=Roles and resource requirements=
Line 86: Line 45:
* "linux/mac/win32 firefox builders"
* "linux/mac/win32 firefox builders"
** builds firefox and pushes for QA
** builds firefox and pushes for QA
** needs 2GB memory, 6GB storage (each)
** l10n/update verification
** needs 2GB memory, 10GB storage (each)
*** e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
*** e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
* "updates builder"
* "updates builder"
Line 99: Line 59:
** e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2
** e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2


=Notes on staging setup=
=Updating Staging release version=
 
* Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. {{bug|407672}}
Bootstrap uses a local CVS mirror, and the "tag", "source", "updates", and "stage" builders are run by a local buildslave.
 
The bootstrap [http://mxr.mozilla.org/mozilla/source/tools/release/Makefile Makefile] has the following targets:
 
* stage/clean_stage
** create/remove basic fileserver/tag/source/updates/stage environment
* cvsmirror/clean_cvsmirror
** create/remove cvsmirror in /builds/cvsmirror
 
These targets are hard-coded to prepare for a 2.0.0.4 release.
 
=Production setup HOWTO for linux/mac/win32=
 
* build-console setup
** use "stage" target in [http://mxr.mozilla.org/mozilla/source/tools/release/Makefile#7 bootstrap's Makefile]
** check out /mofo/release/stage to /data/cltbld/bin
 
* (Win32/Mac only) install Config::General
** FIXME - this should not be necessary, it's not actually used on these platforms
  cd /tools/dist
  wget http://search.cpan.org/CPAN/authors/id/T/TL/TLINDEN/Config-General-2.33.tar.gz
  tar xfvz Config-General-2.33.tar.gz
  cd Config-General-2.33
  perl Makefile.PL
 
its ok to ignore the warning from "perl Makefile.PL":
Warning: the following files are missing in your kit:
t/test.rc.out
 
  sudo make install
 
* (Linux only) prepend custom GCC to the path in ~/.bash_profile
export PATH="/usr/gcc-3.3.2rh/bin:/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:$PYTHONHOME/bin:$PATH"
 
* create logs dir
$ mkdir -p /tools/dist/logs
$ mkdir -p /builds/logs
 
* look for Tinderbox directory
#linux: if tinderbox name is not "Fx-Mozilla1.8-Release" exactly, symlink it
ln -s /builds/tinderbox/Fx-Mozilla1.8-release /builds/tinderbox/Fx-Mozilla1.8-Release
 
* set up Tinderbox l10n build directory
# linux
cd /builds/tinderbox/
# win32
cd /cygdrive/c/builds/tinderbox/
 
mkdir Fx-Mozilla-1.8-l10n-Release
cd Fx-Mozilla-1.8-l10n-Release
../mozilla/tools/tinderbox/install-links
rm build-seamonkey.pl
ln -s ../mozilla/tools/tinderbox/build-firefox.pl .
ln -s build-firefox.pl build-seamonkey.pl
rm post-mozilla.pl
ln -s post-mozilla-release.pl post-mozilla.pl
 
Check out tinderbox configs:
# win32
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/win32
# linux
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -r MOZILLA_1_8_BRANCH_l10n_release -d tinderbox-configs mozilla/tools/tinderbox-configs/firefox/linux
 
ln -s tinderbox-configs/mozconfig .
ln -s tinderbox-configs/tinder-config.pl .
 
* [http://wiki.mozilla.org/ReferencePlatforms/BuildBot/Linux Install buildbot]
* running as "cltbld", install slave
#linux
$ cd ~
$ buildbot create linux-slave1 build-console.build.mozilla.org:9989 linux-slave1 password
#win32
c:\\buildtools\\python24\\scripts\\buildbot create-slave c:\\win32-slave1 build-console.build.mozilla.org:9989 win32-slave1 password
 
* edit the admin and host pages in ~/linux-slave1/info/
 
* start slave
#linux
buildbot start /home/cltbld/linux-slave1
# win32
c:\\buildtools\\python24\\scripts\\buildbot start c:\\win32-slave1
 
==Just for testing==
 
* Move prod ssh keys out of the way, and copy in "staging" keys:
cd ~
mv ~/.ssh ~/ssh.prod
scp cltbld@staging-prometheus-vm:~/.ssh/id_rsa .ssh/
 
* Move prod tinderbox-configs and put staging-build-console in Root:
# win32
cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-Release
 
cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot
vi tinderbox-configs/CVS/Root
 
Same for l10n tinderbox build directories:
 
# win32
cd /cygdrive/c/builds/tinderbox/Fx-Mozilla-1.8-l10n-Release
# linux
cd /builds/tinderbox/Fx-Mozilla-1.8-l10n-Release
 
cp -rp tinderbox-configs tinderbox-configs.prod
# change root to cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot 
vi tinderbox-configs/CVS/Root
 
=Production changes=
==Changing roles==
 
# move to dedicated machines, e.g. production-prometheus-vm
# CVS tag on linux slave or on build-console? ''build-console''
# l10nverify on mac slave ''ok, need to fix "unpack all xpis bug"''
# Mac - is identical hardware req'd? What happens if prod hardware dies? fireball still worked on, scarce PPC hardware options.
 
Available PPCs:


* 01 - head node
* As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.
* 02 - production
* 03 - given to community
* 04 - 1.8.0
* 05 - dead
* 06 - given to community
* fireball - unknown


''discussed: planned switch to Intel''
* As cltbld@staging-build-console, refresh cvsmirror by doing:
cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror


==Staging/Production Buildbot master differences==
* Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).


# Signing - prod waits for signed bits, stage fakes w/ symlink ''ok''
As buildmaster@staging-build-console:
# Bootstrap - prod pulls tag e.g. RELEASE_AUTOMATION_M5, staging pulls tip ''ok''
# pull configs and restart buildbot
cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`


==Outstanding issues==
As cltbld@staging-build-console:
# update mofo scripts
cd /data/cltbld/bin
cvs up


# How to handle bootstrap logs.. remove them between runs? Don't want accumulation on slaves ''remove at start''
* Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
# How to do mock release.. fake version (e.g. 1.2.3.4)? Early 2.0.0.7, that we know we won't release? ''2007 rc1''
# "Source" and "Staging" steps - install a buildslave on stage, or stage everything on build-console? ''use build-console''
# Make sure QA checks e.g. top 5 extensions after Mac Intel switch


=Caveats=
# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux


==Manual steps==
For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.


NOTE - manual steps should be done in this order
* re-enable cltbld's nightly cronjob


# bootstrap configuration
=Setup notes (historical)=
# kicking off buildbot ("buildbot sendchange ...")
* [[Build:Release Automation:Notes on Staging Setup|Notes on Staging Setup]]
# update verification config (working on this in [https://bugzilla.mozilla.org/show_bug.cgi?id=373995 bug 373995]. For now, need to modify and check in the [http://mxr.mozilla.org/mozilla/source/testing/release/updates/moz18-firefox-linux.cfg appropriate] [http://mxr.mozilla.org/mozilla/source/testing/release/updates/moz18-firefox-mac.cfg update] [http://mxr.mozilla.org/mozilla/source/testing/release/updates/moz18-firefox-win32.cfg configs], after all en-US builds but before updates
* [[Build:Release Automation:Notes on Production Setup|Notes on Production Setup]]
# win32 signing, after win32 l10n repack but before updates
* [[Build:Release Automation:Trunk|Notes on Trunk Automation Setup]]
# final installer signing

Latest revision as of 20:39, 17 March 2017

Warning: This RelEng page is obsolete!
It was last edited in 2008, and is thus certainly out of date by now.

Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Buildbot calls bootstrap, parallelizing and serializing where needed.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Buildbot

Buildbot is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.

We have a vendor branch in mozilla/tools/buildbot, based on Buildbot's 0.7.5 release.

Mozilla-specific Buildbot install instructions

Buildbot user manual

Current status

This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the Releases page.

All configs are checked into CVS under mozilla/tools/buildbot-configs/automation.

Roles and resource requirements

  • buildbot master
    • keeps logs, manages overall process
  • ftp/stage.m.o
    • fileserver, both public and private areas
    • FTP candidates - 20GB storage
    • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
    • FTP private staging - 20GB storage
      • e.g. stage:firefox-2.0.0.4/
    • FTP release - 6GB storage
      • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
  • "tagging" builder
    • checks out source and applies tag
    • 2GB storage
      • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
  • "source archive" builder
    • builds source archive and pushes for QA
  • "linux/mac/win32 firefox builders"
    • builds firefox and pushes for QA
    • l10n/update verification
    • needs 2GB memory, 10GB storage (each)
      • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
  • "updates builder"
    • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
    • updates - 1GB memory, 5GB storage
    • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
  • "stage builder"
    • creates private staging area on FTP, renames files for release
    • see "fileserver" requirements, above
  • Automatic Update Server (AUS), aus2.m.o
    • 10GB for config files, backups and staging area
    • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Updating Staging release version

  • Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. bug 407672
  • As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.
  • As cltbld@staging-build-console, refresh cvsmirror by doing:
cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror
  • Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).

As buildmaster@staging-build-console: 

# pull configs and restart buildbot
cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`

As cltbld@staging-build-console: 

# update mofo scripts
cd /data/cltbld/bin
cvs up
  • Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.
# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux

For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.

  • re-enable cltbld's nightly cronjob

Setup notes (historical)

Retrieved from "https://wiki.mozilla.org/index.php?title=Build:Release_Automation&oldid=1166043"