Confirmed users
4,293
edits
ReleaseEngineering/BuildAPI: Difference between revisions
m
→BuildAPI: formatting
m (→BuildAPI: formatting) |
|||
| (30 intermediate revisions by 7 users not shown) | |||
| Line 1: | Line 1: | ||
{{Release Engineering How To|BuildAPI}} | |||
= Overview = | = Overview = | ||
The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs. | The BuildAPI is a Pylons project used by RelEng to surface information collected from two databases updated through our buildbot masters as they run jobs. | ||
| Line 8: | Line 9: | ||
* [http://pypi.python.org/pypi/MySQL-python/ MySQL-Python] | * [http://pypi.python.org/pypi/MySQL-python/ MySQL-Python] | ||
* [https://code.google.com/p/google-visualization-python/ Google-visualization-python] | * [https://code.google.com/p/google-visualization-python/ Google-visualization-python] | ||
* [http:// | * [http://memcached.org/ Memcached] | ||
* Snapshots of our two databases 'schedulerdb' and 'statusdb'. They are available on Cruncher | * Snapshots of our two databases 'schedulerdb' and 'statusdb'. They are available on Cruncher | ||
| Line 14: | Line 15: | ||
If you are looking to setup a local virtual environment to run BuildAPI, then follow [[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|this wiki]]. | If you are looking to setup a local virtual environment to run BuildAPI, then follow [[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|this wiki]]. | ||
= Setup = | |||
Mana documentation: | |||
* [https://mana.mozilla.org/wiki/display/IT/Redis Redis] | |||
* [https://mana.mozilla.org/wiki/display/IT/BuildAPI BuildAPI] | |||
* [https://mana.mozilla.org/wiki/display/IT/RelEng+RabbitMQ] | |||
Puppet setup: | |||
* [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent selfserve_agent] | |||
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/buildapi buildapi] | |||
* [http://hg.mozilla.org/build/puppet-manifests/file/default/modules/redis redis] | |||
= Getting Started = | = Getting Started = | ||
The source for the buildapi is available here: [http://hg.mozilla.org/build/buildapi buildapi source] | The source for the buildapi is available here: [http://hg.mozilla.org/build/buildapi buildapi source] | ||
| Line 136: | Line 147: | ||
= Maintenance = | = Maintenance = | ||
== Updating code == | == Updating code == | ||
BuildAPI has two components: BuildAPI and Selfserve Agent. They're deployed differently | |||
=== BuildAPI === | |||
BuildAPI updates are handled like any other webapp, through pushes. See also [[ReleaseEngineering/How_To/Update_BuildAPI]] | |||
The steps are roughly as follows, from the base directory of buildapi: | |||
* Push new version of BuildAPI to HG | |||
** Edit setup.py to show the updated version (eg version='0.3.0' becomes version='0.3.1') | |||
* Build a new sdist tarball of BuildAPI, using normal python techniques, and upload it to the our pypi repository on relengwebadm.private.scl3.mozilla.com. | |||
** Tarballs can be seen here: http://pypi.pub.build.mozilla.org/pub/ | |||
python setup.py sdist | |||
VERSION=0.3.1 | |||
scp dist/buildapi-${VERSION}.tar.gz relengwebadm.private.scl3.mozilla.com: | |||
ssh relengwebadm.private.scl3.mozilla.com | |||
VERSION=0.3.1 | |||
sudo mv buildapi-${VERSION}.tar.gz /mnt/netapp/relengweb/pypi/pub/ | |||
sudo chmod 664 /mnt/netapp/relengweb/pypi/pub/buildapi-${VERSION}.tar.gz | |||
* Deploy to the webheads, reusing the relengwebadm ssh session from above | |||
sudo su - | |||
cd /data/releng/src/buildapi # or for staging /data/releng-stage/src/buildapi | |||
./update 0.3.1 | |||
=== Selfserve Agent === | |||
Deploying new functionality may require deploying a new version of the <code>scripts/selfserve_agent.py</code> file on build masters. This process is handled by puppet, but orthogonal to buildapi itself. The steps are roughly: | |||
* bump the version of the buildapi repository in "<code>setup.py</code>" and commit (following semantic versioning guidelines, please) | |||
* add a new tag to the buildapi repository for that version, commit & push | |||
* create a python sdist tarball of that version: | |||
** make sure you have a clean checkout | |||
** locally modify (but do not commit) <code>setup.cfg</code> and comment out "<code>tag_build</code>" | |||
** run "python setup.py "<code>sdist</code>" | |||
** updload the file from "<code>dist/</code>" | |||
* [[ReleaseEngineering/How_To/Add_a_Python_Package_to_PuppetAgain|upload to the internal puppetagain python]] repo. | |||
* update the manifests for the new version in [http://hg.mozilla.org/build/puppet/file/default/modules/selfserve_agent/manifests/install.pp puppet] | |||
* wait for puppet to deploy the new agent. | |||
== Adding branches == | == Adding branches == | ||
| Line 142: | Line 192: | ||
== Kicking == | == Kicking == | ||
If things are broken, figure out if it's | |||
* self-serve requests not completing (check the self-serve agents and rabbit connections) | |||
* buildapi HTTP requests not working (check buildapi - [[ReleaseEngineering/How_To/Restart_BuildAPI]]) | |||
* builds-4hr.js not being updated (check the [[ReleaseEngineering/BuildAPI#report_crontask|report crontask]] report crontask) | |||
=== report crontask === | |||
There's a crontask on [https://mana.mozilla.org/wiki/display/websites/Releng+Cluster relengwebadm] that runs every minute to generate the builds-4hr.js file. If this crontask gets "hung", it will prevent updates from occuring. This can happen due to DB connection issues, or bad data in the DB. In these cases, killing the hung crontask is the appropriate fix. There may be a stale lockfile that need removing under /var/lock/buildapi too. | |||
However, the crontask takes about 15 minutes to run on a cold cache. So if its cache (memcached) has gone cold, then killing it before 15 minutes have elapsed is only delaying the failure. | |||
From [https://bugzilla.mozilla.org/show_bug.cgi?id=1005342#c6 bug 1005342]: | |||
{| class="wikitable" | |||
|- | |||
| ''Dustin J. Mitchell [:dustin] 2014-05-02 16:18:14 PDT'' | |||
I looked at 4:01 pacific, and saw that builds-4hr.js.gz had last been generated at 2:47. This is generated by a crontab that runs every minute, and makes a lot of heavy queries against the DB. On a good day with a cold cache, it's about 15 minutes. With a hot cache it can finish comfortably in one minute. It would seem that Matt's heavy query caused the builds-4hr.js.gz queries to go slowly, and thus caused the file's generation to fall behind. | |||
I found the crontask, appropriately dated 2:47, and killed it. Cron started a new one up on the minute, and that completed in something like 45s. | |||
I don't know why the original job didn't complete soon after mpressman cancelled his query. | |||
|} | |||
= Staging = | = Staging = | ||
You can visit https://secure-pub-build.allizom.org/buildapi | |||
From Dustin: | |||
Buildapi itself doesn't write anything but jobrequests. For the staging instance, those go to a different DB than for production. Like the production version, it reads from the production status and scheduler db's. | |||
The bit that it's missing is a running selfserve-agent. If you trigger some action on the staging instance, it pushes a message to the staging rabbitmq virtualhost and waits for a response, but since there's no agent that response never comes. | |||
= Development environment = | |||
Running your own instance of BuildAPI is relatively simple. You should: | Running your own instance of BuildAPI is relatively simple. You should: | ||
* Create a virtualenv | * Create a virtualenv ([[ReleaseEngineering/BuildAPI/Setup_Local_Virtualenv_for_BuildAPI|details]]) | ||
* Install buildapi with its setup.py script | * Install buildapi with its setup.py script | ||
* Create a config. You may base your config on the production one, with at least the following changes: | * Create a config. You may base your config on the production one, with at least the following changes: | ||
| Line 156: | Line 236: | ||
= Troubleshooting Tips = | = Troubleshooting Tips = | ||
== Host == | |||
NOTE: buildapi01 is largely unused now and the hosts are managed by webops | |||
==No MySQLdb== | ==No MySQLdb== | ||
If after installing you run: | If after installing you run: | ||