WebDev/Deployments

This covers deploying of production systems using pip and PyPI.

Goals

• to use pip and Python Package Index (PyPI) whenever possible
• to allow any contributor to easily install and contribute WebDev projects
• to have them deployed quickly and securely internally

Using requirements for a deployment

For a deployed product (such as a website) specify its packages in requirements files. We recommend splitting the packages down into separate file for separate roles, for example: requirements/dev.text, requirements/prod.txt, and requirements/test.txt (those needed only for CI builds).

All requirements should be pinned, e.g.:

  foo==0.3

Use of >= or not pinning to a version is not recommended. This could result in deploying broken, untested versions of packages to production or during local development.

When running pip use the flag: --no-deps. This ensures that packages will not pull in more untested versions of packages. It also means that the requirements files are a definitive source of packages used. This allows security faster audits of who is using what package.

Note: Packages on PyPI can be removed then replaced so even pinning a version does not mean it can be trusted if you are not using the internal mirror.

Using requirements for a library

When building a library for deployed products to install, we recommend creating a single requirements.txt file in the root of the project. Unlike deployed products, do not pin requirements for libraries. Instead, keep them loose like:

   django
   django-aesfield

If you require certain versions, specify the minimum version required:

   django>=1.4

Using loose requirements for libraries helps to avoid version conflicts when a deployed product has multiple packages that both require django (or some common package).

Internal package server

When a package is to be used, it will be uploaded to the internal package server.

Existing servers:

• For add-ons and marketplace pyrepo1.

The trusted users who can upload apps to the server are responsible for validating the packages before uploading them. It's up to the uploading user to validate that the package meets security needs.

TODO add in notes about how to upload.

Contributors and external users will continue to use PyPI as normal. Internal production deployments will use this mirror by using the --no-index and --find-links. For example in Marketplace:

   ./venv/bin/pip install --exists-action=w --no-deps --no-index 
   -f http://pyrepo1.addons.phx1.mozilla.com/ -r requirements/prod.txt

Future goals

If we can build RPMs prior to deploying, this will allow security to more easily audit the source of files. Currently services and socorro do this.