The Mozilla implementation of PuppetAgain has as one of its goals that any client can communicate freely with any master. Ordinarily, each puppetmaster has its own certificate authority, and a client which has been issued a certificate by one master will not be recognized by another master.

The solution is certificate chaining. This section aims to include enough detail to help other puppet users reproduce this configuration elsewhere.

Certificate Hierarchy

We have a hierarchy of certificate authorities that looks like this:

      +-puppetmaster1 CA--+-puppetmaster1 server cert
      |                   |
      |                   +-client 1 server cert
base--+                   :
      |                   
      +-puppetmaster2 CA--+-puppetmaster2 server cert
                          |
                          +-client 10 server cert
                          :

Master Initialization

When a master is initialized, it is provided with a puppetmaster CA certificate and corresponding key. This certificate is signed by the base CA. It immediately creates a "leaf" certificate, suitable for authenticating an SSL connection, signed by this puppetmaster CA certificate.

Certificate Issuance

When a client is issued a new certificate, after the requisite authentication has been performed (see Puppetization Process), a new certificate and key are generated and signed by the CA on the puppetmaster doing the issuing. Any puppetmaster can issue a certificate. This certificate and key, along with the base CA certificate, are returned to the client.

Apache Configuration and Validation

The relevant Apache configuration looks like this:

   SSLCertificateFile /var/lib/puppet/ssl-master/certs/puppet.pem
   SSLCertificateKeyFile /var/lib/puppet/ssl-master/private_keys/puppet.pem
   # include the intermediate cert (this host's CA)
   SSLCertificateChainFile /var/lib/puppet/ssl-master/ca/ca_crt.pem

   # certs and CRLs for all CAs (the base, plus the CA for each puppet
   # master) are stored in the same dir.  This dir is synchronized with csync2.
   SSLCACertificatePath /var/lib/puppet/ssl-master/certdir
   SSLCARevocationPath /var/lib/puppet/ssl-master/certdir

   # verification is optional; the puppet rails app looks at SSL_CLIENT_VERIFY
   # and auth.conf to decide what requires verification
   SSLVerifyClient optional

   # puppetagain cert chains have two CA's, so verify twice the depth
   SSLVerifyDepth  2
   SSLOptions +StdEnvVars

   # The following client headers allow the same configuration to work with Pound.
   RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
   RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
   RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

The certdir contains all puppetmaster CA certs and their CRL's, as well as the base CA cert and its CRL. These are hashed using the following script:

 for i in *.crl; do
   h=`openssl crl -hash -noout -in $i`
   fn=$h.r0
   [ ! -f $fn ] && ln -s $i $fn
 done
 for i in *.crt; do
   h=`openssl x509 -hash -noout -in $i`
   fn=$h.0
   [ ! -f $fn ] && ln -s $i $fn
 done

Note that this directory is only consulted by Apache when validating the client certificates.

CRLs

Certificate revocation lists (CRLs) are crucial to puppet's operation, as certificates are often re-issued to clients, e.g., when they are re-imaged.

All of the CA's described above have corresponding CRLs, and these are made available to Apache with the SSLCARevocationPath directive, as seen above.

Unfortunately, until 14550 is fixed, the puppet agent cannot validate a certificate chain's CRLs, so CRL checking is disabled on the agent:

 certificate_revocation = false

Validation

Remember, Puppet validates certificates in both directions. Let's start with the agent validating the master -- the more common direction for HTTPS. The agent already has the base CA's certificate (provided during certificate issuance, above), and is configured to trust it and anything it signs. The SSLCertificateFile and SSLCertificateKeyFile options point to the puppetmaster's leaf certificate ("puppetmaster1 server cert" in the ascii-art above) and its key. The SSLCertificateChainFile contains the puppetmaster's CA certificate, and Apache includes this in the SSL setup transaction. Thus, the agent has all three certificates needed to validate the master's certificate, and does so. Some care is needed here to make sure that the necessary extensions are present on all of these certificates, as described below.

The master also validates the agent's certificate. The client presents only its client certificate, so Apache must look up the parent certificates in the certdir. This is critical, since the client certificate may have been signed by a different master. After following this certificate chain, Apache consults any CRLs present in the certdir, as well, and refuses access to any revoked certificate, or any certificate signed by a revoked CA certificate. Apache will only follow two levels of chained certificates (SSLVerifyDepth).

Well, that's not quite true: SSLVerifyClient optional means that Apache will run the puppetmaster app even if client validation fails. The three RequestHeader directives pass information on the validation results to puppet, which decides how to handle the request (based, in part, on auth.conf, which is left at its default settings).

ReleaseEngineering/PuppetAgain/Certificate Chaining

Contents