Personal tools

ReleaseEngineering/Mozpool

From MozillaWiki

Jump to: navigation, search

Contents

Overview

Mozilla needs to run its applications on various mobile devices, such as Tegras, Pandas, and even full smartphones. These devices do not act much like the servers that fill the rest of Mozilla's datacenters: they have limited resources, no redundancy, and are comparatively unreliable. With the advent of Firefox OS, Mozilla also needs the ability to automatically reinstall the entire OS on devices.

Mozpool is a system for managing these devices. Users (automated or human) who need a device matching certain specifications can request one from Mozpool, and Mozpool will find such a device, installing a new operating system if necessary. The middle layer of the system (Lifeguard) handles such reinstalls reliably, and also detects and investigates device failure, removing problematic devices from the pool. System administrators can examine these failed devices and repair them, returning them to the pool. The lowest level, Black Mobile Magic (BMM), handles low-level hardware details: automatic power control via IP-addressable power switches; a network-hosted Linux environment for performing software installations; and pinging, logging, and so forth.

Because continued operation of this system is business-critical, it is designed to be resilient to failure not only of individual devices, but to the servers running Mozpool itself.

Policies and Procedures

Available Device Images

panda-android-4.0.4_v3.2
Added SUTAgent 1.20 to base image
panda-android-4.0.4_v3.3
Added Adobe flash 11.1.115.81 to base image
panda-android-4.0.4_v3.1
todo
android
todo
repair-boot
todo
b2g
obsolete

How-To's

Links

Architectural Description

See http://hg.mozilla.org/build/mozpool/file/default/README.md for the most up-to-date architectural description of the system.

Source

The source is at http://hg.mozilla.org/build/mozpool

User Interface

The Mozpool user interface is available through a web browser. The home page shows the three layers of the system (Mozpool, Lifeguard, and BMM). Clicking on any of those shows a UI specific to the layer. The BMM UI allows direct control of device power, as well as manual PXE booting; this layer is of most interest to datacenter operations staff. The lifeguard layer allows managed PXE boots and power cycles, as well as forced state transitions.

Deployment

Mozpool is a Python daemon that runs on multiple imaging servers. It uses a database backend and HTTP API for communication between servers. Its frontend is a dynamic web application. The BMM equipment - TFTP servers, syslog daemons, and so on - runs on the same systems.

Mozpool is designed to be deployed in multiple "pools" within Mozilla. The first and likely largest is release engineering.

Release Engineering

In the scl1 datacenter, we have an initial deployment of 10 racks of Pandaboards. Each rack holds about 80 Pandas, grouped in custom-built chassis, for a total of about 800 pandas. Each rack also contains seven "foopies" (proxying between pandas and Buildbot) and one imaging server. Each rack has a dedicated VLAN, keeping most network traffic local to the rack. The database backend is MySQL. See the puppet modules, linked above, for more details of the deployment.

At the BMM and Lifeguard levels, each imaging server is responsible for the pandas in its rack, as assigned in inventory. At the Mozpool level, each imaging server is responsible for all requests that were initiated locally. Mozpool uses HTTP to communicate with Lifeguard on other imaging servers when it needs to reserve a non-local device.

Mozpool Client

In Release Engineering we use the mozpool client to talk with the Mozpool servers to request panda boards. To do this we install the python package inside of a virtual environment. The package is stored in here: http://puppetagain.pub.build.mozilla.org/data/python/packages

To create a new packaged version, checkout the mozpool repo and do the following:

  1. Make your code changes
  2. Update the version in setup.py
  3. Add a new line to CHANGES.txt with the new version, the date and what is changing
  4. cd mozpoolclient && python setup.py sdist

To deploy to our pypi setup follow these instructions.

Ask Callek if this fork needs updating: http://hg.mozilla.org/build/tools/lib/python/vendor/mozpoolclient-0.1.4 If that is the case you can do this:

cd tools/lib/python/vendor
hg move mozpoolclient-old_version mozpoolclient-new_version
rsync --recursive --delete ../../../../mozpool/mozpoolclient/* mozpoolclient-new_version
#Bump the version in here http://mxr.mozilla.org/build/source/tools/lib/python/vendorlibs.pth

If you're the pypi package maintainer (armenzg or dustin), you can follow these [ instructions].