User:Djmitche/New Releng Puppet Infrastructure

This is a complete re-implementation of puppet for release engineering.

Goals

• A modern puppet installation, completely specifying all releng infrastructure (including Windows)
• Manifests structured to apply settings across all machines, rather than distinct sets of manifests for each slave silo
• Usable by external parties, both inside and outside of mozilla
• Hands-free installations

General

The releng puppet masters are managed by IT (in fact, managed by IT's puppet infrastructure).

We will be using the same puppet versions as the rest of Mozilla. Currently, this is Puppet-2.7.1. As IT upgrades, the masters will be upgraded; releng can then upgrade the clients using puppet itself.

Manifests

The manifests are currently in http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet, but this will move to http://hg.mozilla.org/build/puppet eventually.

Big Files

Some files are too big to end up in version control -- mostly package sources.

Concerns

• Reproducibility - we should have a source-of-record for all files, and this should be enforced
  • [TODO] how?
• Security - some of the packages have special licensing, or may have other security concerns; these will need to be protected from unauthorized access
• Stability - releng systems should not dynamically update, even for security updates. We will need to have "snapshots" of constantly-evolving repositories like EPEL or updates, and a way to update those on demand.
• Storage - this can wind up taking a *huge* amount of space. Saving space with hard-links between identical files might help, especially with snapshots.
  • [TODO] Is there a utility to fix this?
• Replication -

RPMs / Yum Repos Repos

(dir structure here, sources)

Masters

There is currently one puppet master, although long-term we will have multiple masters.

The masters update their manifests from mercurial once every 5 minutes, with a bit of "splay" added (so it does not always occur on the 5-minute mark). Any errors during the update are emailed, as well as a diff of the manifests when they change; the latter forms a kind of change control.

Cert Signing

 A sysadmin asked the Architect,
   "What's the best way to install a new system?"
 The Architect answered,
   "Turn it on."
 The sysadmin was enlightened.

All of our installation tools are scriptable. These tools are responsible for fetching a signed certificate from the puppet master and installing it on the client before its first boot. This transaction will be authenticated using a protected shared secret. Non-Mozilla users can simply omit this part of the setup and sign certificates by hand.

Clients

Client images should proceed automatically from imaging to a fully-operational state. The base image for each OS is an OS install plus whatever minimal equipment is required to invoke puppet, which is responsible for making the system operational.

I've begun work on this for CentOS 6.0 in User:Djmitche/CentOS-60 Base Image.