MozillaWiki

ReleaseEngineering/How To/Setup buildbot masters in AWS

< ReleaseEngineering‎ | How To
Revision as of 22:21, 16 June 2015 by ChrisCooper (talk | contribs)

This page describes how to set-up a new Buildbot Master in AWS.

Production masters

For buildbot masters that are intended to be doing production builds, tests, etc.

Support files, Wikis

Update production-masters.json in tools. Make sure to not enable the masters before it added to DNS and set up by puppet.

Puppet

  • Make sure your masters are listed in PuppetAgain manifests.
    • you can use a little helper to generate snippets (use the "--help" option to see options to generation other snippets useful below):
python json2pp.py bm118
# or even better:
hg -R tools export $you_commit |grep '"name":' | grep ^+ | awk -F: '{print $2}' | awk -F'"' '{print $2}' | xargs python json2pp.py
  • To create a master instance use the following snippet as a template. Don't forget to adjust the name and region accordingly (or use the snippets produced with the "--bash FILE" option):
ssh buildduty@aws-manager1.bb.releng.scl3.mozilla.com
source /builds/aws_manager/bin/activate
cd /builds/aws_manager

#get an ip in the correct region
ip=`python cloud-tools/scripts/free_ips.py -c cloud-tools/configs/buildbot-master -r us-west-2 -n1`
# double-check that the IP address is not in use by some other machine
host $ip
# create a DNS entry
# use full LDAP e.g. user@mozilla.com
invtool A create --ip $ip --target buildbot-master118.bb.releng.usw2.mozilla.com --private --description "bug #: bm118"
# create a DNS reverse-mapping (required for puppet certs to work properly)
invtool PTR create --ip $ip --target buildbot-master118.bb.releng.usw2.mozilla.com --private --description "bug #: bm118"
#create a CNAME
invtool CNAME create --ip $ip --target buildbot-master118.build.mozilla.org --private --description "bug #: bm118"
sleep 20m # wait for DNS to propagate

#create the instance
python cloud-tools/scripts/aws_create_instance.py -c cloud-tools/configs/buildbot-master -r us-west-2 -s aws-releng \
-k secrets/aws-secrets.json --ssh-key ~/.ssh/aws-ssh-key  \
-i cloud-tools/instance_data/us-west-2.instance_data_master.json buildbot-master118

Puppet will reboot the the master. You can follow the log in a different terminal: 

tail -F buildbot-master118.log
  • You can get the IP address of the master from the log or using AWS web console.
  • When master rebooted make sure to login as cltbld and stop the master:
cd /builds/buildbot/*1* && make stop
  • For build/try masters, add master's ip to secrets::network::masterIPs on master-puppet1:/etc/puppet/manifests/secrets.pp. The signing instances will be automatically reloaded.

Add masters to inventory

Create System in Inventory

As of 2013-05-13, there are two ways, manual data entry and CSV import

CSV Import

Manual Entry

  • Go to https://inventory.mozilla.org/en-US/systems/new/
  • Fill the following fields:
    • Hostname: use FQDN
    • System Status: production
    • System Rack: Releng-AWS-VPC - Releng-AWS-USE1/USW2
    • System Type: Virtual Server
    • Operating System: Centos 6
    • Allocated To: release
  • Click "Create"

Common Setup

  • Edit the new created entry
  • Switch to "Key/Value Store"
  • Click "Add adapter"
  • Fill the following fields:
    • Adapter Number (0-99): 0
    • IP Address: IP address
    • Mac Address: output of ip link show eth0 | tail -n -1 | awk '{print $2}' on the master
    • Adapter Name (nic0-99 or mgmt0-99): nic0
    • Host Name: fqdn
    • Leave the rest as is
  • Press "Create", then close box, to dismiss the adapter dialog
  • Press "Save" to save the adapter (even though adapter may not show)

Add masters to slavealloc

See Adding your master to slavealloc

Add master's SSH key to known_hosts

See bug 889992

  • Add the master's ssh key to known_hosts in puppet (to make release-runner work):
    • dump the key by running the following:
ssh-keyscan $master_name
    • verify the change (ssh to masters as cltbld from bm81)
ssh -i .ssh/release-runner buildbot-master80.bb.releng.usw2.mozilla.com

IT

  1. File a Operations Server Operations bug to:
    • add the master(s) to Nagios (See bug 861359)
      • take note of the checks that are requested in the sample bug
      • you should be able to see the masters in here: use1 and usw2

Questions

  • I have not been able to find a bug where we clearly request for adding hosts/IPs to the "Build Masters group"
  • It is unclear why some masters don't need the flow to releng-rabiitmq (see comment)
  • Should we talk with NetOps to document steps to add a master?

Lock a slave and let it take jobs

  • Locked through slavealloc a slave to the newly setup master.
  • Let it run for a couple of hours and check that the jobs worked well.
  • Check also in the #buildduty channel for possible nagios checks going off (e.g. Queue directories checks)

If no issues are found then go ahead and enable the master on slavealloc.

Final Verification

The following steps should be performed to ensure the rest has worked okay (all steps should be run as user cltbld):

  • SSH verification (and associated netflows).
for h in ffxbld trybld xrbld tbirdbld; do ssh -i ~/.ssh/${h}_dsa $h@stage.mozilla.org id ; done
for h in ffxbld; do ssh -i ~/.ssh/${h}_dsa $h@pvtbuilds2.dmz.scl3.mozilla.com id ; done
  • mySQL verification (associated netflows):
nc -zv buildbot-rw-vip.db.scl3.mozilla.com 3306
  • ensure nagios checks are all green and notifications are enabled (aka not disabled), eg
http://nagios1.private.releng.scl3.mozilla.com/releng-scl3/cgi-bin/status.cgi?navbarsearch=1&host=buildbot-master43
Retrieved from "https://wiki.mozilla.org/index.php?title=ReleaseEngineering/How_To/Setup_buildbot_masters_in_AWS&oldid=1080517"