ReleaseEngineering/Applications/Slavealloc: Difference between revisions

Jump to navigation Jump to search

ReleaseEngineering/Applications/Slavealloc (view source)

Revision as of 21:09, 10 July 2012

4,810 bytes removed ,  10 July 2012
no edit summary
No edit summary
Line 12: Line 12:
** Twisted
** Twisted
** SQLAlchemy
** SQLAlchemy
== External Resources ==
The slavealloc server uses the following external resources:
* two MySQL databases (production and staging)
* VM host for the slaveallocator VM
== Security ==
The slave allocator hands out low-security slave passwords in the .tac files, which are stored in cleartext in the database.  It does not do any sort of authentication either for read or modify operations, and relies on the Build VLAN firewalls to prevent external access.
== Monitoring ==
The slavealloc host has the basic host monitoring from nagios (ping, filesystems, etc.), plus an HTTP GET to /api/pools, just to make sure the daemon is still responding.
== Deployment ==
The slave allocator server is deployed on a single host, <tt>slavealloc.build.mozilla.org</tt>.
=== Server Setup ===
IT installed RHEL6 along with MySQL client libraries, and set up the proper firewall rules to allow database access.
As root, virtualenv-1.5.2 was installed into the system Python library.  The following system packages were installed via yum:
* http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm (for EPEL packages; use --nogpgcheck)
* hg
* MySQL-python
* nginx (from EPEL repo)
* nscd (and enable it)
=== Nginx ===
Nginx frontends for both staging and production instances.  The virtualhosts files are available [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib in hg].
Note that on the x86_64 system slavealloc is currently installed on, the following must be added to the <tt>http</tt> section of <tt>nginx.conf</tt>:
    # required to use virtualhosts - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=547722
    server_names_hash_bucket_size 33;
=== Twisted Daemon ===
The 'slavealloc' user runs the twisted daemon on this host.  The user account is locked and accessed only via su from root.
The daemon is installed in a virtualenv at <tt>/tools/slavealloc-$rev</tt>, using the pre-checked python packages on the puppet server.  Note that <tt>--no-site-packages</tt> is ''not'' used here, because we need access to the (binary) MySQL-python package which is installed systemwide:
cd /tools
virtualenv slavealloc-8fe4dbc09d03
./slavealloc/bin/pip install -e hg+http://hg.mozilla.org/build/tools@8fe4dbc09d03#egg=tools \
    --no-index --find-links=http://staging-puppet.build.mozilla.org/staging/python-packages/
ln -s slavealloc-8fe4dbc09d03 slavealloc
ln -s slavealloc-8fe4dbc09d03 slavealloc-staging
There is a [http://hg.mozilla.org/build/tools/file/tip/lib/python/slavealloc/contrib/make-slavealloc-virtualenv.sh make-slavealloc-virtualenv.sh] script available in <tt>/tools</tt> to make this process automatic.
Once this was set up, the 'slavealloc dbinit' command was used to initialize the database. 
The production and staging tac files are in <tt>/build/slavealloc</tt>.  Staging runs on port 1079, and production runs on 1080.  The files are similar to those [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib in hg], and include the commented <tt>pool_recycle</tt> line, with timeout 400, to automatically expire connections before the MySQL server itself does.  If this still occurs, we can look to the example of the [https://github.com/djmitche/buildbot/blob/master/master/buildbot/db/enginestrategy.py#L31 Buildbot source] for a better solution.
Startup is done via [http://hg.mozilla.org/build/tools/file/828895e6a5d3/lib/python/slavealloc/contrib/ initscripts].
=== Slave Side ===
All slaves run [http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildslave/files/runslave.py runslave.py] during startup.  This file is distributed via puppet. The larger slave-startup process is described in [[ReleaseEngineering/Buildslave Startup Process]].
=== Backups ===
The slavealloc server has
11 4 * * * slavealloc /tools/slavealloc/bin/slavealloc dbdump -D mysql://mumblemumblemumble > /builds/slavealloc/production-1080/dbdump.pkl
in <tt>/etc/cron.d/slavealloc-bkup</tt>, as a basic protection against someone accidentally running dbinit or doing something equally catastrophic.
== Staging ==
As described above, http://staging-slavealloc.build.mozilla.org/ points to a staging implementation of the slave allocator.  This implementation has its own database runs from a distinct daemon, although it is served from the same nginx instance.
'''NOTE:''' ''all'' slaves are configured use the production allocator.  Allocations from the staging allocator will need to be simulated by hand (<tt>runslave.py</tt> has an command-line option to set the allocator URL).  This is done to allow us, as a group, to move slaves between staging and production using a single slave allocator.
The [http://hg.mozilla.org/build/tools/file/tip/lib/python/slavealloc/contrib/prod-db-to-staging.sh prod-db-to-staging.sh] script will copy the production db to staging for use when staging new changes.


== Development ==
== Development ==
Line 93: Line 27:


== See Also ==
== See Also ==
See [[User:Djmitche/Slave Allocator Proposal]]
* See [[User:Djmitche/Slave Allocator Proposal]]
* See https://mana.mozilla.org/wiki/display/IT/Slavealloc for more detail on the setup at Mozilla.
Djmitche
canmove, Confirmed users
1,394

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/449330"

Navigation menu