Build Masters

we have lots!

The user-visible list is at Masters (auth required). The canonical list is http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json

source for buildbot is at http://hg.mozilla.org/build/buildbot/

Clobberer

https://build.mozilla.org/clobberer/ and https://build.mozilla.org/stage-clobberer

Source is at http://hg.mozilla.org/build/tools/file/default/clobberer

DB is a sqlite file local to the web server

Buildapi

includes self-serve, wait time reports, pending/running

Source Code

http://hg.mozilla.org/build/buildapi

Public Interfaces

https://build.mozilla.org/buildapi

Current Setup

production instance is running on cruncher.b.m.o out of ~catlee/buildapi-official. There is a cron job that checks for new changes every 5 minutes and deploys them. This instance is exposed to the public behind LDAP auth via proxy from the build.m.o url.

The buildapi instance communicates with a rabbitmq server also running on cruncher. It also connects to two read-only databases (schedulerdb, statusdb) on a slave mysql host maintained by IT.

The self-serve agent, which is responsible for performing many of the job requests, is running on production-master03 out of ~cltbld/buildapi. It is a long-running process that waits for messages from the rabbitmq server on cruncher, and then makes various DB modifications or pokes at masters' web interfaces. The self-serve agent is started via init, and monitored via nagios.

To modify the list of branches supported by buildapi and self-serve, read ReleaseEngineering/BuildAPI#Maintenance

Setup Instructions

Described at ReleaseEngineering/BuildAPI

slavealloc

(still being deployed .. so at the moment all of this is future)

Application Description

Slavealloc is a client-server application. The client is runslave.py. Communication is via a very basic HTTP request to http://slavealloc.build.mozilla.org/production/gettac/$slavename, where the response is expected to be a buildbot.tac file suitable for use to start buildslave.

The slavealloc server is a implemented as a small Twisted application (source) which serves the tac generator, a JSON REST interface, and a client-side JavaScript interface.

The same source code also implements a command-line frontend to the REST interface.

Requirements

Deployment

The slave allocator server is deployed on a single host, slavealloc.build.mozilla.org.

Server Setup

IT installed RHEL6 along with MySQL client libraries, and set up the proper firewall rules to allow database access.

As root, virtualenv-1.5.2 was installed into the system Python library. The following system packages were installed via yum:

hg
MySQL-python

Twisted Daemon

The 'slavealloc' user runs the twisted daemon on this host. The user account is locked and accessed only via su from root.

The daemon is installed in a virtualenv at /tools/slavealloc-$rev, using the pre-checked python packages on the puppet server:

cd /tools
virtualenv --no-site-packages slavealloc-8fe4dbc09d03
/slavealloc-8fe4dbc09d03/bin/pip install -e hg+http://hg.mozilla.org/build/tools@8fe4dbc09d03#egg=tools \
   --no-index --find-links=http://staging-puppet.build.mozilla.org/staging/python-packages/

External Resources

Security

The slave allocator hands out low-security slave passwords in the .tac files, which are stored in cleartext in the database. It does not do any sort of authentication either for read or modify operations, and relies on the Build VLAN firewalls to prevent external access.

Development

Install tools in a virtualenv:

cd tools
virtualenv sandbox
sandbox/bin/pip install -e .

Then you can run the slavealloc daemon locally from the root of the tools repository with a simple:

sandbox/bin/twistd -noy slavealloc-combined.tac

Note that due to what I believe to be a bug in pip, you may need to explicitly install Twisted to get the twistd executable installed:

pip install -U twisted

This configuration will use SQLite to access {{{slavealloc.db}}} in the current directory. You can hack on the static web content while the daemon is running.

regression detection

code is http://hg.mozilla.org/graphs/file/9c02347596b4/server/analysis

runs on cruncher via cronjob under catlee's account from ~/graphs/server/analysis

deployment questions

what languages are needed and their version
- if python, what python modules - can they be run in a virtualenv
- if perl, what cpan modules are needed
- if php, what php version and what php.ini entries are needed
what is the command line parameters to start the web service
does it have any special configuration or init files
does it require root or sudo
does it require a special directory layout
will it generate/use temp files or non-database assets?
what version of mysql and what database config, where is the sql to init the tables
does memcached have a min/max memory? which processes read/write to it - can it be on different IP
what ports will be opened for listening
cronjobs that need to be run
if outside services are utilized, what is that list

maintenance questions

what are common issues we run into, and how to debug?
- place in buildduty docs?
where is the code/schema for hacking/reading?
are there special passwords/accounts/acl's, and what are they? (not in public docs)

ReleaseEngineering/Applications

Contents