ReleaseEngineering/Applications: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(removing obsolete applications)
 
(26 intermediate revisions by 10 users not shown)
Line 1: Line 1:
= Build Masters =
This page is an incomplete attempt to categorize all of the applications (or systems, or infrastructures, or whatever you'd like to call them) installed and in use by release engineering.
we have lots!


The user-visible list is at [https://intranet.mozilla.org/RelEngWiki/index.php/Masters Masters] (auth required).  The canonical list is http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json
= App Store =
* [[ReleaseEngineering/Applications/Ship It]]
* [[ReleaseEngineering/Applications/Tooltool]]
* [[ReleaseEngineering/Applications/RelengAPI]]
* [[Balrog]]


source for buildbot is at http://hg.mozilla.org/build/buildbot/
= What the heck? =
We're not quite sure yet what these pages should contain.  Here are some questions that should be answered for each application:


= Clobberer =
== Deployment Questions ==
https://build.mozilla.org/clobberer/ and https://build.mozilla.org/stage-clobberer
 
Source is at http://hg.mozilla.org/build/tools/file/default/clobberer
 
DB is a sqlite file local to the web server
 
= Buildapi =
includes self-serve, wait time reports, pending/running
 
== Source Code ==
http://hg.mozilla.org/build/buildapi
 
== Public Interfaces ==
https://build.mozilla.org/buildapi
 
== Current Setup ==
production instance is running on cruncher.b.m.o out of ~catlee/buildapi-official. There is a cron job that checks for new changes every 5 minutes and deploys them. This instance is exposed to the public behind LDAP auth via proxy from the build.m.o url.
 
The buildapi instance communicates with a rabbitmq server also running on cruncher. It also connects to two read-only databases (schedulerdb, statusdb) on a slave mysql host maintained by IT.
 
The self-serve agent, which is responsible for performing many of the job requests, is running on production-master03 out of ~cltbld/buildapi. It is a long-running process that waits for messages from the rabbitmq server on cruncher, and then makes various DB modifications or pokes at masters' web interfaces. The self-serve agent is started via init, and monitored via nagios.
 
To modify the list of branches supported by buildapi and self-serve, read [[ReleaseEngineering/BuildAPI#Maintenance]]
 
== Setup Instructions ==
Described at [[ReleaseEngineering/BuildAPI]]
 
= slavealloc =
(still being deployed .. so at the moment all of this is future)
== Application Description ==
Slavealloc is a client-server application.  The client is [http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildslave/files/runslave.py runslave.py].  Communication is via a very basic HTTP request to http://slavealloc.build.mozilla.org/production/gettac/$slavename, where the response is expected to be a <tt>buildbot.tac</tt> file suitable for use to start buildslave.
 
The slavealloc server is a implemented as a small Twisted application ([http://hg.mozilla.org/build/tools/file/tip/slavealloc source]) which serves the tac generator, a JSON REST interface, and a client-side JavaScript interface.
 
The same source code also implements a command-line frontend to the REST interface.
 
== Requirements ==
 
== Deployment ==
The slave allocator server is deployed on a single host, <tt>slavealloc.build.mozilla.org</tt>.
 
=== Server Setup ===
IT installed RHEL6 along with MySQL client libraries, and set up the proper firewall rules to allow database access.
 
As root, virtualenv-1.5.2 was installed into the system Python library.  The following system packages were installed via yum:
* hg
* MySQL-python
 
=== Twisted Daemon ===
The 'slavealloc' user runs the twisted daemon on this host.  The user account is locked and accessed only via su from root.
 
The daemon is installed in a virtualenv at <tt>/tools/slavealloc-$rev</tt>, using the pre-checked python packages on the puppet server:
cd /tools
virtualenv --no-site-packages slavealloc-8fe4dbc09d03
/slavealloc-8fe4dbc09d03/bin/pip install -e hg+http://hg.mozilla.org/build/tools@8fe4dbc09d03#egg=tools \
    --no-index --find-links=http://staging-puppet.build.mozilla.org/staging/python-packages/
 
== External Resources ==
 
== Security ==
The slave allocator hands out low-security slave passwords in the .tac files, which are stored in cleartext in the database.  It does not do any sort of authentication either for read or modify operations, and relies on the Build VLAN firewalls to prevent external access.
 
== Development ==
Install tools in a virtualenv:
cd tools
virtualenv sandbox
sandbox/bin/pip install -e .
Then you can run the slavealloc daemon locally from the root of the tools repository with a simple:
sandbox/bin/twistd -noy slavealloc-combined.tac
Note that due to what I believe to be a bug in pip, you may need to explicitly install Twisted to get the twistd executable installed:
pip install -U twisted
 
This configuration will use SQLite to access {{{slavealloc.db}}} in the current directory.  You can hack on the static web content while the daemon is running.
 
== See Also ==
See [[User:Djmitche/Slave Allocator Proposal]]
 
= regression detection =
code is http://hg.mozilla.org/graphs/file/9c02347596b4/server/analysis
 
runs on cruncher via cronjob under catlee's account from ~/graphs/server/analysis
 
=deployment questions=
* what languages are needed and their version
* what languages are needed and their version
** if python, what python modules - can they be run in a virtualenv
** if python, what python modules - can they be run in a virtualenv
Line 96: Line 17:
* what is the command line parameters to start the web service
* what is the command line parameters to start the web service
* does it have any special configuration or init files
* does it have any special configuration or init files
** How are those backed up?
* does it require root or sudo
* does it require root or sudo
* does it require a special directory layout
* does it require a special directory layout
* will it generate/use temp files or non-database assets?
* will it generate/use temp files or non-database assets?
** which of those need to be backed up?
** how is that done?
* what version of mysql and what database config, where is the sql to init the tables
* what version of mysql and what database config, where is the sql to init the tables
** what is the backup strategy?
* does memcached have a min/max memory? which processes read/write to it - can it be on different IP
* does memcached have a min/max memory? which processes read/write to it - can it be on different IP
* what ports will be opened for listening
* what ports will be opened for listening
Line 105: Line 30:
* if outside services are utilized, what is that list
* if outside services are utilized, what is that list


=maintenance questions=
== Maintenance Questions ==
* what are common issues we run into, and how to debug?
* what are common issues we run into, and how to debug?
** place in buildduty docs?
** place in buildduty docs?
* where is the code/schema for hacking/reading?
* where is the code/schema for hacking/reading?
* are there special passwords/accounts/acl's, and what are they? (not in public docs)
* are there special passwords/accounts/acl's, and what are they? (not in public docs)

Latest revision as of 21:50, 19 November 2018

This page is an incomplete attempt to categorize all of the applications (or systems, or infrastructures, or whatever you'd like to call them) installed and in use by release engineering.

App Store

What the heck?

We're not quite sure yet what these pages should contain. Here are some questions that should be answered for each application:

Deployment Questions

  • what languages are needed and their version
    • if python, what python modules - can they be run in a virtualenv
    • if perl, what cpan modules are needed
    • if php, what php version and what php.ini entries are needed
  • what is the command line parameters to start the web service
  • does it have any special configuration or init files
    • How are those backed up?
  • does it require root or sudo
  • does it require a special directory layout
  • will it generate/use temp files or non-database assets?
    • which of those need to be backed up?
    • how is that done?
  • what version of mysql and what database config, where is the sql to init the tables
    • what is the backup strategy?
  • does memcached have a min/max memory? which processes read/write to it - can it be on different IP
  • what ports will be opened for listening
  • cronjobs that need to be run
  • if outside services are utilized, what is that list

Maintenance Questions

  • what are common issues we run into, and how to debug?
    • place in buildduty docs?
  • where is the code/schema for hacking/reading?
  • are there special passwords/accounts/acl's, and what are they? (not in public docs)