Personal tools

Labs/Bespin/ProductionDeployment

From MozillaWiki

Jump to: navigation, search

Contents

Production Deployment

This document describes the production deployment of the Python backend. The Python backend is a standard WSGI application, which means that there are numerous ways to deploy it. There are even standalone WSGI servers that you can deploy it with, including one that comes with Paste and Spawning.

Building

Bespin uses Paver to handle its builds. The top-level pavement file knows how to package up Bespin for a production deployment. At Bespin's top-level directory, run the following command to build Bespin:

paver dist

This command will compress the JavaScript, put together a Python package and produce a complete tarball that is ready to be untarred on the production server. The final tarball lives in build/BespinServer.tar.gz.

Installing

Copy the tarball built in the previous step to your production server. It will go into the home directory of the account that you'll be running the app as. We use mod_wsgi's daemon model (vs. in-process model), which allows us to run Bespin as a different user than the web server.

untar the file, and you will end up with a BespinServer directory. In there, run:

python bootstrap.py --no-site-packages

This will set up a virtualenv and then install all of the dependencies. This installation procedure requires that the production server have outbound access to the internet to grab the packages required by Bespin. If the production server does not have outbound access, you can look into packaging up the dependencies into a single collection. We're using pip to install the dependencies, and there are some options to help collect up the requirements.

  • as of 5/20/09 2 packages path and Paste had conflicts with pip install. Manually install path and install it:
bin/pip install http://pypi.python.org/packages/source/p/path.py/path-2.2.zip

the latest release of Paste didn't include the required patches for bespin. Manually copy over Paste from your dev system to production system under <mybespindirectory>/libs and install:

bin/pip install libs/Paste-1.7.3dev-r7791.tar.gz

Apache with mod_wsgi

We chose Apache with mod_wsgi because that combination will automatically manage the Python processes for us. We are using mod_wsgi 2.3. It is not uncommon for people to Apache or nginx with a proxy to a Python web server.

Here is the Bespin part of our Apache configuration with comments inline to describe the different parts of the config:

 # gzip all of our plaintext resources to make downloads faster
 AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript application/x-javascript
 
 # give Apache access to the directory where we will store the Bespin WSGI script
 <Directory /home/WSGIUSER/wsgi-apps>
 Order allow,deny
 Allow from all
 </Directory>
 
 # Optional, give Apache access to the static files so that it can serve them directly
 <Directory /home/WSGIUSER/BespinServer/frontend>
 Order allow,deny
 Allow from all
 </Directory>
 
 # our production setup includes a caching load balancer in front.
 # we tell the load balancer to cache for 5 minutes.
 # we can use the commented out lines to tell it to not cache,
 # which is useful if we're updating.
 <FilesMatch "\.(html|js|png|jpg|css)$">
 ExpiresDefault "access plus 5 minutes"
 ExpiresActive On
 #Header unset cache-control:
 #Header append cache-control: "no-cache, must-revalidate"
 </FilesMatch>
 
 # tell Apache where to find our index file. We sometimes will put up
 # a temporary message while doing maintenance
 #AliasMatch ^/$ /home/WSGIUSER/BespinServer/frontend/temporary.html
 AliasMatch ^/$ /home/WSGIUSER/BespinServer/frontend/index.html
 
 # here's the mod_wsgi configuration. We're setting up 2 processes, each
 # with 15 threads. Note that the root of the server is served by
 # bespin.wsgi. The WSGIPythonHome points to a vitualenv directory
 # into which all of the libraries are installed.
 WSGIScriptAlias / /home/WSGIUSER/wsgi-apps/bespin.wsgi
 WSGIDaemonProcess bespin user=WSGIUSER processes=2 threads=15
 WSGIProcessGroup bespin
 WSGIPythonHome /home/WSGIUSER/BespinServer
 
 WSGISocketPrefix run/wsgi
 
 # serve up the static files directly from Apache.
 # The python web server *can* serve the static files, but Apache
 # is much more efficient at it.
 Alias /images/ /home/WSGIUSER/BespinServer/frontend/images/
 Alias /css/ /home/WSGIUSER/BespinServer/frontend/css/
 Alias /js/ /home/WSGIUSER/BespinServer/frontend/js/
 #Alias /index.html /home/WSGIUSER/BespinServer/frontend/temporary.html
 #Alias /editor.html /home/WSGIUSER/BespinServer/frontend/temporary.html
 #Alias /dashboard.html /home/WSGIUSER/BespinServer/frontend/temporary.html
 Alias /index.html /home/WSGIUSER/BespinServer/frontend/index.html
 Alias /editor.html /home/WSGIUSER/BespinServer/frontend/editor.html
 Alias /dashboard.html /home/WSGIUSER/BespinServer/frontend/dashboard.html
 Alias /docs/code/ /home/WSGIUSER/BespinServer/frontend/js/
 Alias /docs/ /home/WSGIUSER/BespinServer/docs/

Most of the configuration above is optional. In fact, the only parts required are:

  1. directory directive to provide access to the wsgi-apps directory (where you put the bespin.wsgi script)
  2. The WSGI* directives that tell mod_wsgi what to run and how to run it.

The WSGI script

mod_wsgi scripts are standard Python scripts that include an "application" object in their namespace. That is the WSGI application that is attached to the server.

Bespin's run time configuration is all stored in variables in the bespin.config module in a dictionary-like object called "c". The WSGI script *could* do all of the itself by directly setting the values. In the script below, we set the values via a simple .ini file. Additionally, we set up Python's logging system to put any log output that interests us into logs/bespin.log.

 import os
 import logging
 import logging.handlers
 
 import ConfigParser
 from bespin import config, controllers
 
 import sys
 #redirect stdout to go to stderr, it will appear in the apache error log
 sys.stdout = sys.stderr
 
 # setting the configuration profile starts us off with reasonable values for our
 # environment.
 config.set_profile("prod")
 
 # load config values from the INI file
 cp = ConfigParser.ConfigParser()
 cp.read(os.path.abspath(os.path.dirname(__file__) + "/../config/bespin.ini"))
 
 # config values are available as a dictionary, and config.c is a dictionary
 # so we can just do config.c.update.
 config.c.update(cp.items("config"))
 
 # set up logging
 root_log = logging.getLogger()
 root_log.setLevel(logging.WARN)
 handler = logging.handlers.RotatingFileHandler(
            os.path.abspath(os.path.dirname(__file__) + "/../logs/bespin.log"))
 root_log.addHandler(handler)
 handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
 
 # email out errors
 handler2 = logging.handlers.SMTPHandler('localhost', 'YOU@YOU.COM',
                                         ['YOU@YOU.COM'],
                                         'Bespin Error')
 handler2.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
 handler2.setLevel(logging.ERROR)
 
 # uncomment the next line if you actually want the errors mailed, the
 # above is sample config
 #root_log.addHandler(handler2)
 
 # activate profile is responsible for taking the config and creating database
 # connections, etc.
 config.activate_profile()
 
 # this is the bit for mod_wsgi. controllers.make_app() gives us the top-level WSGI
 # application.
 application = controllers.make_app()


If you don't plan to run a full asynchronous setup on the server (requiring you to install beanstalkd and run bespin_worker processes), you'll want to set this flag *before* the config.activate_profile() in the WSGI script.

 config.c.async_jobs=False

The .ini file

The INI file is where your site-specific configuration will go.

 [config]
 dburl=mysql://user:password@localhost/bespin
 secret=YOUR SECRET USED IN HASHING
 static_dir=/home/WSGIUSER/BespinServer/frontend
 fsroot=YOUR HOME DIRECTORY FOR ALL BESPIN FILES

Creating the database

Assuming bespin.wsgi is in a directory called wsgi-apps that is at the same level as your BespinServer directory, you can then run

   paver create_db

To set up your production database.