ReleaseEngineering/How To/Setup a buildbot master

This page describes how to set-up a new Buildbot Master.

Production masters

For buildbot masters that are intended to be doing production builds, tests, etc.

Hardware

• Current policy is one buildbot master instance per VM
• 64-bit guest
• 2 virtual CPUs
• 6 GB RAM
• 6 GB swap
• 30GB partition mounted at /
• 100MB partition mounted at /boot

OS

• Install CentOS 5.5
• Make sure hostname is correct is set to the fully qualified name. This can be modified in /etc/sysconfig/network and the hostname command.
• Install puppet

rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
yum install puppet

• (optional - recommended if this isn't "just another box install" for any reason) Run puppetd manually until no further work is required. (If you don't have/need your own puppent environment on the puppet master, just remove that option below)

# get the hostname & key issues out of the way - repeat until no more
# "warning: peer certificate won't be verified in this SSL session" messages.
# (you'll need to do the master-puppet1 signing listed below)
puppetd --test --server master-puppet1.build.scl1.mozilla.com --environment YOUR_ENV --noop
# now work on actual config issues
# start in directory accessible to all users (not '/root')
cd /tmp
puppetd --test --server master-puppet1.build.scl1.mozilla.com --environment YOUR_ENV 
# repeat above until you get no actions taken or error reported

• Configure daemon to point to master-puppet1.build.mozilla.org (regardless of datacenter master resides in)

vim /etc/sysconfig/puppet
PUPPET_SERVER=master-puppet1.build.scl1.mozilla.com
chkconfig puppet on
/etc/init.d/puppet start

# On master-puppet1:
puppetca --sign your-new-master.build.scl1.mozilla.com

Support files, Wikis

Update production-masters.json in tools.

Puppet manifests

• Make sure your masters are listed in buildmaster-production.pp

When you're ready, update the manifests on the master with:

hg -R /etc/puppet/manifests pull
hg -R /etc/puppet/manifests update

Once the manifests are updated the masters' build dirs should be automatically created.

• For build masters, add master's ip to secrets::network::masterIPs on master-puppet1:/etc/puppet/manifests/secrets.pp. The signing instances will need to be reloaded. See [1]

Add masters to slavealloc

See Adding your master to slavealloc

Note: it would be good to know when we flip the enabled bit in slavealloc

IT-related things

File separate bugs for Nagios (eg: bug 717804), Mysql access (eg: bug 717806)

• Nagios
  • PING
  • Swap
  • avg load
  • buildbot
  • disk - /
  • disk - /builds
• Mysql access to the DB server
• Verify that master can send mail to tinderbox via dm-mail01. see e.g. bug 717808

SSH Keys

• Copy production ssh keys (for ffxbld, trybld, xrbld and tbirdbld) and known_hosts to ~/.ssh.
• Make sure you can ssh to ffxbld@dm-pvtbuild01.mozilla.org and ffxbld@stage.mozilla.org (without prompting for "unknown host").

Final Verification

We don't "burn in" buildbot masters - they will go directly to their assigned roles. The following steps should be performed to ensure the rest has worked okay:

• SSH verification (and associated netflows):

    $ ssh -i ~/.ssh/ffxbld_dsa ffxbld@dm-pvtbuild01.mozilla.org hostname --fqdn
    dm-pvtbuild01.mozilla.org
    $ ssh -i ~/.ssh/ffxbld_dsa ffxbld@stage.mozilla.org hostname --fqdn
    surf.mozilla.org

• mySQL verification (and associated netflows):

    $ mysql -h buildbot-ro-vip.db.scl3.mozilla.com
    ERROR 1045 (28000): Access denied for user 'cltbld'@'10.22.70.209' (using password: NO)
    $ mysql -h buildbot-rw-vip.db.scl3.mozilla.com
    ERROR 1045 (28000): Access denied for user 'cltbld'@'10.22.70.209' (using password: NO)

Personal / development masters

See ReleaseEngineering/How To/Setup Personal Development Master