ReleaseEngineering/How To/Setup buildbot masters in AWS

This page describes how to set-up a new Buildbot Master in AWS.

Production masters

For buildbot masters that are intended to be doing production builds, tests, etc.

Support files, Wikis

Update production-masters.json in tools. Make sure to not enable the masters before it added to DNS and set up by puppet.

Puppet

• Make sure your masters are listed in PuppetAgain manifests.
  • you can use a little helper to generate snippets (use the "--help" option to see options to generation other snippets useful below):

python json2pp.py bm58
# or even better:
hg -R tools export $you_commit |grep '"name":' | grep ^+ | awk -F: '{print $2}' | awk -F'"' '{print $2}' | xargs python json2pp.py

• To create a master instance use the following snippet as a template. Don't forget to adjust the name and region accordingly (or use the snippets produced with the "--bash FILE" option):

ssh cruncher.build.mozilla.org
sudo su - buildduty
source ~/aws/aws-ve/bin/activate
cd ~/aws/cloud-tools/aws
hg pull -u
eval `ssh-agent`
ssh-add secrets/aws-releng
python aws_create_instance.py -c configs/master-linux64 -r us-west-2 -s aws-releng  \
  -k /home/buildduty/aws/cloud-tools/aws/secrets/aws-secrets.json \
  -i us-west-2.instance_data_master.json buildbot-master58

Puppet will reboot the the master. You can follow the log in a different terminal:

tail -F buildbot-master58.log

• You can get the IP address of the master from the log or using AWS web console.
• When master rebooted make sure to login as cltbld and stop the master:

cd /build/buildbot/build1 && make stop

• For build/try masters, add master's ip to secrets::network::masterIPs on master-puppet1:/etc/puppet/manifests/secrets.pp. The signing instances will be automatically reloaded.

Add masters to inventory

Create System in Inventory

As of 2013-05-13, there are two ways, manual data entry and CSV import

CSV Import

• Go to https://inventory.mozilla.org/en-US/csv/
• Import the file created with the "--inventory-csv FILE" option above.
• After import, use the links in the results page to visit each server & set allocation to "release"
• Continue with common setup below

Manual Entry

• Go to https://inventory.mozilla.org/en-US/systems/new/
• Fill the following fields:
  • Hostname: use FQDN
  • System Status: production
  • System Rack: Releng-AWS-VPC - Releng-AWS-USE1/USW2
  • Operating System: Centos 6
  • Allocated To: release
• Click "Create"

Common Setup

• Edit the new created entry
• Switch to "Key/Value Store"
• Click "Add adapter"
• Fill the following fields:
  • Adapter Number (0-99): 0
  • IP Address: IP address
  • Mac Address: output of ip link show eth0 | tail -n -1 | awk '{print $2}' on the master
  • Adapter Name (nic0-99 or mgmt0-99): nic0
  • Leave the rest as is

Add masters to slavealloc

See Adding your master to slavealloc

Add master's SSH key to know_hosts

• Add the master's ssh key to known_hosts (to make release-runner work):
  • dump the key by running the following:

ssh-keyscan $master_name

• • ad it to /N/production/home/cltbld/.ssh/known_hosts on master-puppet1.build.scl1.mozilla.com
  • add it to bm36:~cltbld/.ssh/known_hosts (puppet doesn't work here)
  • TODO: add steps for puppetagain
  • verify the change (ssh to masters as cltbld from bm36)

ssh -i .ssh/release-runner buildbot-master80.srv.releng.usw2.mozilla.com

IT

• Ask RelOps to add the master to DNS once you added it to inventory (for example, bug 859524)
• Ask NetOps (ACL Requests) to add new hosts to the "Build Masters group" (for example bug 859528, with update)
• Nagios (bug 861359)

Lock a slave and let it take jobs

• Locked through slavealloc a slave to the newly setup master.
• Let it run for a couple of hours and check that the jobs worked well.
• Check also in the #buildduty channel for possible nagios checks going off (e.g. Queue directories checks)

If no issues are found then go ahead and enable the master on slavealloc.

Final Verification

The following steps should be performed to ensure the rest has worked okay (all steps should be run as user cltbld):

• SSH verification (and associated netflows).

for h in ffxbld trybld xrbld tbirdbld; do ssh -i ~/.ssh/${h}_dsa $h@stage.mozilla.org id ; done
for h in ffxbld; do ssh -i ~/.ssh/${h}_dsa $h@pvtbuilds2.dmz.scl3.mozilla.com id ; done

• mySQL verification (associated netflows):

nc -zv buildbot-rw-vip.db.scl3.mozilla.com 3306

• ensure nagios checks are all green and notifications are enabled (aka not disabled), eg

http://nagios1.private.releng.scl3.mozilla.com/releng-scl3/cgi-bin/status.cgi?navbarsearch=1&host=buildbot-master43