ReleaseEngineering/Applications: Difference between revisions

Jump to navigation Jump to search

ReleaseEngineering/Applications (view source)

Revision as of 16:25, 22 March 2011

4,913 bytes removed ,  22 March 2011
no edit summary
No edit summary
Line 1: Line 1:
= Build Masters =
This page is an incomplete attempt to categorize all of the applications (or systems, or infrastructures, or whatever you'd like to call them) installed and in use by release engineering.
we have lots!


The user-visible list is at [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (auth required).  The canonical list is http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json
= App Store =
* [[Applications/Buildbot]]
* [[Applications/Clobberer]]
* [[Applications/BuildAPI]]
* [[Applications/Slavealloc]]
* [[Applicatoins/Regression Detection]]


source for buildbot is at http://hg.mozilla.org/build/buildbot/
= What the heck? =
We're not quite sure yet what these pages should contain.  Here are some questions that should be answered for each application:


= Clobberer =
== Deployment Questions ==
https://build.mozilla.org/clobberer/ and https://build.mozilla.org/stage-clobberer
 
Source is at http://hg.mozilla.org/build/tools/file/default/clobberer
 
DB is a sqlite file local to the web server
 
= Buildapi =
includes self-serve, wait time reports, pending/running
 
== Source Code ==
http://hg.mozilla.org/build/buildapi
 
== Public Interfaces ==
https://build.mozilla.org/buildapi
 
== Current Setup ==
production instance is running on cruncher.b.m.o out of ~catlee/buildapi-official. There is a cron job that checks for new changes every 5 minutes and deploys them. This instance is exposed to the public behind LDAP auth via proxy from the build.m.o url.
 
The buildapi instance communicates with a rabbitmq server also running on cruncher. It also connects to two read-only databases (schedulerdb, statusdb) on a slave mysql host maintained by IT.
 
The self-serve agent, which is responsible for performing many of the job requests, is running on production-master03 out of ~cltbld/buildapi. It is a long-running process that waits for messages from the rabbitmq server on cruncher, and then makes various DB modifications or pokes at masters' web interfaces. The self-serve agent is started via init, and monitored via nagios.
 
To modify the list of branches supported by buildapi and self-serve, read [[ReleaseEngineering/BuildAPI#Maintenance]]
 
== Setup Instructions ==
Described at [[ReleaseEngineering/BuildAPI]]
 
= slavealloc =
(still being deployed .. so at the moment much of this is future)
== Application Description ==
Slavealloc is a client-server application.  The client is [http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildslave/files/runslave.py runslave.py].  Communication is via a very basic HTTP request to http://slavealloc.build.mozilla.org/production/gettac/$slavename, where the response is expected to be a <tt>buildbot.tac</tt> file suitable for use to start buildslave.
 
The slavealloc server is a implemented as a small Twisted application ([http://hg.mozilla.org/build/tools/file/tip/lib/python/slavealloc source]) which serves the tac generator, a JSON REST interface, and a client-side JavaScript interface.
 
The same source code also implements a command-line frontend to the REST interface.
 
== Requirements ==
 
== Deployment ==
The slave allocator server is deployed on a single host, <tt>slavealloc.build.mozilla.org</tt>.
 
=== Server Setup ===
IT installed RHEL6 along with MySQL client libraries, and set up the proper firewall rules to allow database access.
 
As root, virtualenv-1.5.2 was installed into the system Python library.  The following system packages were installed via yum:
* http://download.fedora.redhat.com/pub/epel/6/i386/epel-release-6-5.noarch.rpm (for EPEL packages; use --nogpgcheck)
* hg
* MySQL-python
* nginx (from EPEL repo)
 
=== Nginx ===
Nginx frontends for both
 
=== Twisted Daemon ===
The 'slavealloc' user runs the twisted daemon on this host.  The user account is locked and accessed only via su from root.
 
The daemon is installed in a virtualenv at <tt>/tools/slavealloc-$rev</tt>, using the pre-checked python packages on the puppet server.  Note that <tt>--no-site-packages</tt> is ''not'' used here, because we need access to the (binary) MySQL-python package which is installed systemwide:
cd /tools
virtualenv slavealloc-8fe4dbc09d03
ln -s slavealloc-8fe4dbc09d03 slavealloc
/slavealloc-8fe4dbc09d03/bin/pip install -e hg+http://hg.mozilla.org/build/tools@8fe4dbc09d03#egg=tools \
    --no-index --find-links=http://staging-puppet.build.mozilla.org/staging/python-packages/
 
Once this was set up, the 'slavealloc dbinit' command was used to initialize the database.
 
The production and staging tac files are in <tt>/build/slavealloc</tt>.  Staging runs on port 1079, and production runs on 1080.
 
TODO (temp)
* initscript
* nginx
* set up production
 
== External Resources ==
 
== Security ==
The slave allocator hands out low-security slave passwords in the .tac files, which are stored in cleartext in the database.  It does not do any sort of authentication either for read or modify operations, and relies on the Build VLAN firewalls to prevent external access.
 
== Development ==
Install tools in a virtualenv:
cd tools
virtualenv sandbox
sandbox/bin/pip install -e .
Then you can run the slavealloc daemon locally from the root of the tools repository with a simple:
sandbox/bin/twistd -noy slavealloc-combined.tac
Note that due to what I believe to be a bug in pip, you may need to explicitly install Twisted to get the twistd executable installed:
pip install -U twisted
To set up a fresh database, use
sandbox/bin/slavealloc dbinit -D sqlite:///slavealloc.db
This configuration will use SQLite to access {{{slavealloc.db}}} in the current directory.  You can hack on the static web content while the daemon is running.
 
== See Also ==
See [[User:Djmitche/Slave Allocator Proposal]]
 
= regression detection =
code is http://hg.mozilla.org/graphs/file/9c02347596b4/server/analysis
 
runs on cruncher via cronjob under catlee's account from ~/graphs/server/analysis
 
=deployment questions=
* what languages are needed and their version
* what languages are needed and their version
** if python, what python modules - can they be run in a virtualenv
** if python, what python modules - can they be run in a virtualenv
Line 121: Line 27:
* if outside services are utilized, what is that list
* if outside services are utilized, what is that list


=maintenance questions=
== Maintenance Questions ==
* what are common issues we run into, and how to debug?
* what are common issues we run into, and how to debug?
** place in buildduty docs?
** place in buildduty docs?
* where is the code/schema for hacking/reading?
* where is the code/schema for hacking/reading?
* are there special passwords/accounts/acl's, and what are they? (not in public docs)
* are there special passwords/accounts/acl's, and what are they? (not in public docs)
Djmitche
canmove, Confirmed users
1,394

edits

Retrieved from "https://wiki.mozilla.org/Special:MobileDiff/293192"

Navigation menu