MozillaWiki

ReleaseEngineering/How To/Setup buildbot masters in AWS

< ReleaseEngineering‎ | How To
Revision as of 00:10, 16 April 2014 by Rail (talk | contribs) (→‎Puppet)

This page describes how to set-up a new Buildbot Master in AWS.

Production masters

For buildbot masters that are intended to be doing production builds, tests, etc.

Support files, Wikis

Update production-masters.json in tools. Make sure to not enable the masters before it added to DNS and set up by puppet.

Puppet

  • Make sure your masters are listed in PuppetAgain manifests.
    • you can use a little helper to generate snippets (use the "--help" option to see options to generation other snippets useful below):
python json2pp.py bm58
# or even better:
hg -R tools export $you_commit |grep '"name":' | grep ^+ | awk -F: '{print $2}' | awk -F'"' '{print $2}' | xargs python json2pp.py
  • To create a master instance use the following snippet as a template. Don't forget to adjust the name and region accordingly (or use the snippets produced with the "--bash FILE" option):
ssh buildduty@aws-manager1.srv.releng.scl3.mozilla.com
source /builds/aws_manager/bin/activate
cd /builds/aws_manager

python scripts/aws_create_instance.py -c configs/master-linux64 -r us-west-2 -s aws-releng  \
  -k secrets/aws-secrets.json --ssh-key ~/.ssh/aws-ssh-key \
  -i cloud-tools/instance_data/us-west-2.instance_data_master.json buildbot-master58

Puppet will reboot the the master. You can follow the log in a different terminal: 

tail -F buildbot-master58.log
  • You can get the IP address of the master from the log or using AWS web console.
  • When master rebooted make sure to login as cltbld and stop the master:
cd /builds/buildbot/*1* && make stop
  • For build/try masters, add master's ip to secrets::network::masterIPs on master-puppet1:/etc/puppet/manifests/secrets.pp. The signing instances will be automatically reloaded.

Add masters to inventory

Create System in Inventory

As of 2013-05-13, there are two ways, manual data entry and CSV import

CSV Import

Manual Entry

  • Go to https://inventory.mozilla.org/en-US/systems/new/
  • Fill the following fields:
    • Hostname: use FQDN
    • System Status: production
    • System Rack: Releng-AWS-VPC - Releng-AWS-USE1/USW2
    • Operating System: Centos 6
    • Allocated To: release
  • Click "Create"

Common Setup

  • Edit the new created entry
  • Switch to "Key/Value Store"
  • Click "Add adapter"
  • Fill the following fields:
    • Adapter Number (0-99): 0
    • IP Address: IP address
    • Mac Address: output of ip link show eth0 | tail -n -1 | awk '{print $2}' on the master
    • Adapter Name (nic0-99 or mgmt0-99): nic0
    • Host Name: fqdn
    • Leave the rest as is
  • Press "Create", then close box, to dismiss the adapter dialog
  • Press "Save" to save the adapter (even though adapter may not show)

Add masters to slavealloc

See Adding your master to slavealloc

Add master's SSH key to known_hosts

See bug 889992

  • Add the master's ssh key to known_hosts in puppet (to make release-runner work):
    • dump the key by running the following:
ssh-keyscan $master_name
    • verify the change (ssh to masters as cltbld from bm81)
ssh -i .ssh/release-runner buildbot-master80.srv.releng.usw2.mozilla.com

IT

  • Ask NetOps (ACL Requests) to add new hosts to the "Build Masters group" (for example bug 859528, with update)
  • Ask NetOps to add new IPs to the "Build Masters" list in the firewall rules (NEW as of 2013-07-12)
  • Ask NetOps (ACL Requests) to open netflows from the master to releng-rabbitmq-zlb.webapp.scl3.mozilla.com 5672 (for example bug 985088) self-serve agent. NEW as of 2014-03-18
  • Ask Server Ops to add the master to Nagios (bug 861359)

Lock a slave and let it take jobs

  • Locked through slavealloc a slave to the newly setup master.
  • Let it run for a couple of hours and check that the jobs worked well.
  • Check also in the #buildduty channel for possible nagios checks going off (e.g. Queue directories checks)

If no issues are found then go ahead and enable the master on slavealloc.

Final Verification

The following steps should be performed to ensure the rest has worked okay (all steps should be run as user cltbld):

  • SSH verification (and associated netflows).
for h in ffxbld trybld xrbld tbirdbld; do ssh -i ~/.ssh/${h}_dsa $h@stage.mozilla.org id ; done
for h in ffxbld; do ssh -i ~/.ssh/${h}_dsa $h@pvtbuilds2.dmz.scl3.mozilla.com id ; done
  • mySQL verification (associated netflows):
nc -zv buildbot-rw-vip.db.scl3.mozilla.com 3306
  • ensure nagios checks are all green and notifications are enabled (aka not disabled), eg
http://nagios1.private.releng.scl3.mozilla.com/releng-scl3/cgi-bin/status.cgi?navbarsearch=1&host=buildbot-master43
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS&oldid=965241"