ReleaseEngineering/BuildAPI/Setup Local Virtualenv for BuildAPI

The purpose of this wiki is to show you how to setup a virtual environment for buildapi and run buildapi inside it.
Note: These instructions are currently only for Mac OSX

Installing/Creating a VirtualEnv

1. Install pip¹ and virtualenv by running:

   sudo easy_install pip
   sudo pip install virtualenv
   

2. Create and activate your own virtualenv, with your desired name (ie <foo>):

   virtualenv <foo>
   cd <foo>
   source bin/activate
   

Installing Dependencies for your Virtual Environment

1. Install MySQL:
   • Download the most recent .dmg for your version of Mac (for me that was 'Mac OS X ver. 10.7 (x86, 64-bit), DMG Archive')
   • Make a free account, then Oracle will ask what you're doing with MySQL (just tell them, it's no secret) and, finally download it and install the .dmg file
2. Create a symbolic link with libmysqlclient.18.dylib inside your virtualenv by running:

   ln -s /usr/local/mysql-5.6.12-osx10.7-x86_64/lib/libmysqlclient.18.dylib /Users/<user>/buildapi-test/lib/python2.7/site-packages/libmysqlclient.18.dylib

3. Add the following lines to <foo>/bin/activate:

   export PATH=$PATH:/usr/local/mysql/bin
   export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/Users/jzeller/buildapi-test/lib/python2.7/site-packages/

4. Install XCode:
   • Go , launch in the App Store and either sign up for a free account or use your existing credentials (It's free, so no stress).
   • Then open XCode, and go to Preferences>Downloads and install 'Command Line Tools'
     • Note 1: If you are sketched out about having your credit card info attached then follow these instructions.
     • Note 2: This should take care of the issues with not having 'clang' or 'gcc' on your machine
5. Install Mercurial, by running:

   pip install mercurial

6. Install the Google Python Visualization Library (version 1.7.1)
   • Note: Based on 'pip freeze' from buildapi01 supplied by catlee
7. Install Redis
8. Create a pip install file by copy/paste'ing the following snippet into a file called 'requirements.txt'
   • Note: This snippet based on 'pip freeze' from buildapi01 supplied by catlee

   Beaker==1.5.4
   FormEncode==1.2.4
   Mako==0.4.1
   MarkupSafe==0.12
   MySQL-python==1.2.3
   Paste==1.7.5.1
   PasteDeploy==1.5.0
   PasteScript==1.7.3
   Pygments==1.4
   Pylons==1.0
   Routes==1.12.3
   SQLAlchemy==0.7.1
   Tempita==0.5.1
   WebError==0.10.3
   WebHelpers==1.3
   WebOb==1.0.8
   WebTest==1.2.3
   amqplib==0.6.1
   anyjson==0.3.1
   carrot==0.10.7
   decorator==3.3.1
   distribute==0.6.14
   nose==1.0.0
   pytz==2011h
   redis==2.4.5
   simplejson==2.1.6
   wsgiref==0.1.2

9. If you haven't already, activate your virtual environment and move into the <foo> directory by running:

   source <foo>/bin/activate
   cd <foo>
   

   • This step ensures that you are installing dependencies ONLY within the virtual environment
10. Install the requirements by running:

    pip install -r requirements.txt

    • Note: If you run across the error "ImportError: No module named pkg_resources", then type this to fix it²:

      curl http://python-distribute.org/distribute_setup.py | python

11. Clone BuildAPI to your local machine:

    hg clone http://hg.mozilla.org/build/buildapi -r dec9d2ddedb715255db9cfce1a890d6a22bc7cd9

Setup Databases

MySQL (from production) - recommended

These directions will get you real (although old) production databases to use in your local BuildAPI instance. Currently zeller has 2 mysql dumps stored on a public folder in his people.mozilla.com directory that you can use, here: people.mozilla.com:jozeller/public/. If they are not there, or you want newer data, you'll need to post a bug in 'Server Operations:Database' to request a current dump from schedulerdb and statusdb. Once you have one of these, you can now follow these directions.

1. Setup empty MySQL databases for schedulerdb and statusdb:
   1. Make sure you have started your MySQL server, which is in System Preferences
   2. Enter MySQL:

      mysql -uroot -p

   3. Create schedulerdb and statusdb:

      create database schedulerdb;
      create database statusdb;

   4. Exit MySQL:

      \q

2. Now load the mysql dumps into schedulerdb and statusdb

1. mysql --verbose -uroot -p schedulerdb < schedulerdb_dump.sql

2. mysql --verbose -uroot -p statusdb < statusdb_dump.sql

   • WARNING: Schedulerdb will take roughly 1.5 hours and statusdb will take almost 24 hours. That is assuming that these are full database dumps like the ones that the database team do dumps for once per day.

SQlite

These directions will get you empty sqlite databases. If you'd like to run from real production data, I suggest setting up mysql with production dumps from schedulerdb and statusdb.

1. Run the following commands from your <foo>/buildapi/ directory:

   cd buildapi
   sqlite3 schedulerdb
   .databases
   .exit
   sqlite statusdb
   .databases
   .exit
   

Setup and Start BuildAPI

1. To get started with the pylons project, make sure you are in the <foo>/buildapi/ directory and inside your virtualenv, then simply run:

   python setup.py develop

   • This handles grabbing pylon project dependencies. You will also need to grab and install the google python visualization library from here.
2. Now generate a config file for your project, by running:

    paster make-config buildapi config.ini
   

3. Now edit that config.ini
   • If you used the MySQL configuration with a production database
     • Edit config.ini to use the correct database URLs.

       # SQLAlchemy database URLs
       # Buildbot's scheduler database
       sqlalchemy.scheduler_db.url = mysql://root:@localhost/schedulerdb
       # Mozilla's status databasea
       sqlalchemy.status_db.url = mysql://root:@localhost/statusdb
       # buildapi specific database
       sqlalchemy.buildapi_db.url = sqlite:///production.db

   • If you used SQLite configuration without a production database
     • Edit config.ini to use the correct database URLs. SQLite databases take the format
       `sqlite:////absolute/path/to/database_name`.
       For example:

       # SQLAlchemy database URLs
       # Buildbot's scheduler database
       sqlalchemy.scheduler_db.url = sqlite:////Users/<username>/<foo>/buildapi/schedulerdb
       # Mozilla's status databasea
       sqlalchemy.status_db.url = sqlite:////Users/<username>/<foo>/buildapi/statusdb
       # buildapi specific database
       sqlalchemy.buildapi_db.url = sqlite:///production.db
       

   • Add the following line below "branches = b1, b2":

     branches_url = http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json

4. Change the port listed under [server:main] to 7000, or some other port that is available, like so:

   port = 7000

   • Note: If you run into the error "socket.error: [Errno 48] Address already in use", then try a new port. This is likely because the port you've chosen really is busy, and is not some wacky error in this case.
5. You should now be all set up now! Running the project locally can be done through:

   paster serve --reload -v config.ini

   Which starts running the buildapi on your local machine. To view it, open up http://localhost:7000. Or, replace 7000 with whichever port you changed to.

References

1. http://www.tlswebsolutions.com/mac-os-x-lion-setting-up-django-pip-virtualenv-and-homebrew/
2. http://stackoverflow.com/questions/7446187/no-module-named-pkg-resources