ReleaseEngineering/How To/Setup a buildbot master: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 1: Line 1:
These instructions are probably incomplete, but should serve as a good starting point for setting up a new buildbot master. If you find something missing or inaccurate, please update!
This page describes how to set-up a new Buildbot Master.


* Create a new directory for your master, e.g. /builds/buildbot/mymaster
= OS =
* Install glibc-2.5-42.el5_4.3 (CentOS 5.0 only)


* Create a virtual environment for your master
= The Master Instance =
First, to get the master directory prepped and cronjobs set-up, do this on the machine:
<pre>
<pre>
virtualenv --distribute --no-site-packages -p /tools/python-2.6.5/bin/python /builds/buildbot/mymaster
# cltbld@wherever
</pre>
BASEDIR=/builds/buildbot/my_master
 
MASTER_NAME=my_master_name # Must be one from ./setup-master.py -8 -l
* Install required packages
PYTHON=/tools/python-2.6.5/bin/python
<pre>
VIRTUALENV=/tools/python-2.6.5/bin/virtualenv
source /builds/buildbot/mymaster/bin/activate
cd /builds/buildbot/mymaster
pip install simplejson
pip install mysql-python
pip install Twisted==10.1.0
pip install sqlalchemy
pip install argparse
pip install pycrypto
pip install pyasn1
pip install pyopenssl==0.10
pip install http://hg.mozilla.org/users/clegnitto_mozilla.com/mozillapulse/archive/tip.tar.bz2
hg clone http://hg.mozilla.org/build/buildbot
(cd buildbot/master; $PYTHON setup.py develop install)
(cd buildbot/slave; $PYTHON setup.py develop install)
hg clone -b production-0.8 http://hg.mozilla.org/build/buildbotcustom
hg clone -b production http://hg.mozilla.org/build/buildbot-configs
hg clone http://hg.mozilla.org/build/tools
</pre>
 
* Add Makefile
<pre>
PWD=$(shell pwd)
BUILDBOT=$(PWD)/bin/buildbot
HG=/tools/python-2.6.5/bin/hg
HG=/tools/python-2.6.5/bin/hg
export PYTHONDONTWRITEBYTECODE=1
hg clone http://hg.mozilla.org/build/buildbot-configs
export PYTHONPATH=/builds/buildbot/mymaster/tools/lib/python
cd buildbot-configs
 
make -f Makefile.setup all BASEDIR=$BASEDIR MASTER_NAME=$MASTER_NAME VIRTUALENV=$VIRTUALENV PYTHON=$PYTHON HG=$HG
start:
        cd master && $(BUILDBOT) start $$PWD
stop:
        cd master && $(BUILDBOT) stop $$PWD
restart:
        cd master && $(BUILDBOT) restart $$PWD
reconfig:
        cd master && $(BUILDBOT) reconfig $$PWD
checkconfig:
        cd master && $(BUILDBOT) checkconfig
update:
        (cd buildbotcustom && $(HG) pull && $(HG) update -C)
        (cd buildbot-configs && $(HG) pull && $(HG) update -C)
        (cd tools && $(HG) pull && $(HG) update -C)
version:
        $(BUILDBOT) --version
</pre>
 
* Setup master
<pre>
mkdir master
cd master
ln -s ../buildbot-configs/config.py .
ln -s ../buildbot-configs/production_config.py localconfig.py
etc. --catlee too tired to fill this in right now. add symlinks and run make checkconfig until it works!
</pre>
</pre>


* Setup cron jobs
Then, update whatever files the Makefile instructs you too with correct passwords, descriptions, etc.
<pre>
# In /home/cltbld/.mymaster_update_from_files.conf
[DEFAULT]
database = mysql://buildbot:password@tm-b01-master01.mozilla.org/buildbot
master = http://fqdn:http_port
name = My master!
 
# In /etc/cron.d/mymaster
MAILTO=release@mozilla.com
 
@hourly cltbld (source /home/cltbld/.bash_profile && /builds/buildbot/mymaster/bin/python tools/buildfarm/maintenance/watch_twistd_log.py -t .mymaster-last-time.txt -n mymaster -f cltbld@`hostname --fqdn` -e release@mozilla.com /builds/buildbot/mymaster/master)
 
15 * * * * cltbld lockfile -60 -r 3 $HOME/lockfile.mymaster_cleanup && (source /home/cltbld/.bash_profile && hg -R tools pull -u && nice -n 19 /builds/buildbot/mymaster/bin/python tools/buildfarm/maintenance/master_cleanup.py -t4 /builds/buildbot/mymaster/master ; rm -f $HOME/lockfile.mymaster_cleanup) >> cleanup.log 2>&1
 
*/10 * * * * cltbld lockfile -60 -r 3 $HOME/lockfile.bbdb 2>/dev/null && (source /home/cltbld/.bash_profile && cd /builds/buildbot/mymaster/master; time nice /builds/buildbot/mymaster/bin/python /builds/buildbot/mymaster/buildbotcustom/bin/update_from_files.py -c /home/cltbld/.mymaster_update_from_files.conf . ; rm -f $HOME/lockfile.bbdb ) >> update.log 2>&1
</pre>


* Update [http://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json production-masters.json] in tools and the [https://intranet.mozilla.org/RelEngWiki/index.php/Masters master list], if appropriate
= Support files, Wikis =
Update [http://hg.mozilla.org/build/tools/file/default/buildfarm/maintenance/production-masters.json production-masters.json] in tools and the [https://intranet.mozilla.org/RelEngWiki/index.php/Masters master list], if appropriate


* Make sure to file an IT bug to get nagios checks added for the master if necessary
= IT-related things =
* Nagios
** PING
** Swap
** avg load
** buildbot
** disk - /
** disk - /builds
* Mysql access to the DB server
* Add master to Tinderbox whitelist


* SSH Keys
= SSH Keys =
** Copy production ssh keys (for ffxbld and trybld) to ~/.ssh.
* Copy production ssh keys (for ffxbld and trybld) to ~/.ssh.
** Make sure you can ssh to ffxbld@dm-pvtbuild01.mozilla.org and ffxbld@stage.mozilla.org
* Make sure you can ssh to ffxbld@dm-pvtbuild01.mozilla.org and ffxbld@stage.mozilla.org


= Other =
* Make sure NTP is running!
* Make sure NTP is running!

Revision as of 16:59, 18 March 2011

This page describes how to set-up a new Buildbot Master.

OS

  • Install glibc-2.5-42.el5_4.3 (CentOS 5.0 only)

The Master Instance

First, to get the master directory prepped and cronjobs set-up, do this on the machine: 

# cltbld@wherever
BASEDIR=/builds/buildbot/my_master
MASTER_NAME=my_master_name # Must be one from ./setup-master.py -8 -l
PYTHON=/tools/python-2.6.5/bin/python
VIRTUALENV=/tools/python-2.6.5/bin/virtualenv
HG=/tools/python-2.6.5/bin/hg
hg clone http://hg.mozilla.org/build/buildbot-configs
cd buildbot-configs
make -f Makefile.setup all BASEDIR=$BASEDIR MASTER_NAME=$MASTER_NAME VIRTUALENV=$VIRTUALENV PYTHON=$PYTHON HG=$HG

Then, update whatever files the Makefile instructs you too with correct passwords, descriptions, etc.

Support files, Wikis

Update production-masters.json in tools and the master list, if appropriate

IT-related things

  • Nagios
    • PING
    • Swap
    • avg load
    • buildbot
    • disk - /
    • disk - /builds
  • Mysql access to the DB server
  • Add master to Tinderbox whitelist

SSH Keys

  • Copy production ssh keys (for ffxbld and trybld) to ~/.ssh.
  • Make sure you can ssh to ffxbld@dm-pvtbuild01.mozilla.org and ffxbld@stage.mozilla.org

Other

  • Make sure NTP is running!
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Setup_a_buildbot_master&oldid=292100"