ReleaseEngineering/PuppetAgain/Certificate Chaining

The Mozilla implementation of PuppetAgain has as one of its goals that any client can communicate freely with any master. Ordinarily, each puppetmaster has its own certificate authority, and a client which has been issued a certificate by one master will not be recognized by another master.

The solution is certificate chaining. This section aims to include enough detail to help other puppet users reproduce this configuration elsewhere.

Certificate Hierarchy

We have a hierarchy of certificate authorities that looks like this:

      +-puppetmaster1 CA--+-puppetmaster1 server cert
      |                   |
      |                   +-client 1 server cert
base--+                   :
      |                   
      +-puppetmaster2 CA--+-puppetmaster2 server cert
                          |
                          +-client 10 server cert
                          :

Master Initialization

When a master is initialized, it is provided with a puppetmaster CA certificate and corresponding key. This certificate is signed by the base CA. It immediately creates a "leaf" certificate, suitable for authenticating an SSL connection, signed by this puppetmaster CA certificate.

Certificate Issuance

When a client is issued a new certificate, after the requisite authentication has been performed (see Puppetization Process), a new certificate and key are generated and signed by the CA on the puppetmaster doing the issuing. Any puppetmaster can issue a certificate. This certificate and key, along with the base CA certificate, are returned to the client.

Validation

The relevant Apache configuration looks like this:

   SSLCertificateFile /var/lib/puppet/ssl-master/certs/puppet.pem
   SSLCertificateKeyFile /var/lib/puppet/ssl-master/private_keys/puppet.pem
   # include the intermediate cert (this host's CA)
   SSLCertificateChainFile /var/lib/puppet/ssl-master/ca/ca_crt.pem
   # certs and CRLs for all CAs (the base, plus the CA for each puppet
   # master) are stored in the same dir.  This dir is synchronized with csync2.
   SSLCACertificatePath /var/lib/puppet/ssl-master/certdir
   SSLCARevocationPath /var/lib/puppet/ssl-master/certdir
   # verification is optional; the puppet rails app looks at SSL_CLIENT_VERIFY
   # and auth.conf to decide what requires verification
   SSLVerifyClient optional
   # puppetagain cert chains have two CA's, so verify twice the depth
   SSLVerifyDepth  2
   SSLOptions +StdEnvVars
   # The following client headers allow the same configuration to work with Pound.
   RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
   RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
   RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

The certdir contains all puppetmaster CA certs and their CRL's, as well as the base CA cert and its CRL. We synchronize this directory among all puppetmasters using csync2. The certificates and CRLs are hashed using the following script:

 for i in *.crl; do
   h=`openssl crl -hash -noout -in $i`
   fn=$h.r0
   [ ! -f $fn ] && ln -s $i $fn
 done
 for i in *.crt; do
   h=`openssl x509 -hash -noout -in $i`
   fn=$h.0
   [ ! -f $fn ] && ln -s $i $fn
 done

Note that this directory is only consulted by Apache when validating the client certificates.

CRLs

Certificate revocation lists (CRLs) are crucial to puppet's operation, as certificates are often re-issued to clients, e.g., when they are re-imaged.

All of the CA's described above have corresponding CRLs, and these are made available to Apache with the SSLCARevocationPath directive, as seen above.

Puppet Configuration

Unfortunately, until 14550 is fixed, the puppet agent cannot validate a certificate chain's CRLs, so CRL checking is disabled on the agent:

 certificate_revocation = false

On the master, you'll need to convince it not to try to do fancy CA-like things:

 ca = false

Validation

Remember, Puppet validates certificates in both directions. Let's start with the agent validating the master -- the more common direction for HTTPS. The agent already has the base CA's certificate (provided during certificate issuance, above), and is configured to trust it and anything it signs. The SSLCertificateFile and SSLCertificateKeyFile options point to the puppetmaster's leaf certificate ("puppetmaster1 server cert" in the ascii-art above) and its key. The SSLCertificateChainFile contains the puppetmaster's CA certificate, and Apache includes this in the SSL setup transaction. Thus, the agent has all three certificates needed to validate the master's certificate, and does so. Some care is needed here to make sure that the necessary extensions are present on all of these certificates, as described below.

The master also validates the agent's certificate. The client presents only its client certificate, so Apache must look up the parent certificates in the certdir. This is critical, since the client certificate may have been signed by a different master. After following this certificate chain, Apache consults any CRLs present in the certdir, as well, and refuses access to any revoked certificate, or any certificate signed by a revoked CA certificate. Apache will only follow two levels of chained certificates (SSLVerifyDepth).

Well, that's not quite true: SSLVerifyClient optional means that Apache will run the puppetmaster app even if client validation fails. The three RequestHeader directives pass information on the validation results to puppet, which decides how to handle the request (based, in part, on auth.conf, which is left at its default settings).

Revocation

When it comes time to revoke a certificate, things can be a bit complicated. Revocation requires access to the CA certificate and key, but that key only exists on the master that owns it. So a method is required to indicate that a particular certificate should be revoked. We accomplish this with a client_cert directory. This directory has a subdirectory for each puppet master, and each such subdirectory has a revoke subdirectory for certificates that should be revoked. The whole thing might look like

client_certs/puppetmaster1.mozilla.com/client1.mozilla.com.crt
client_certs/puppetmaster1.mozilla.com/client3.mozilla.com.crt
client_certs/puppetmaster1.mozilla.com/revoke/098350a8f.0
client_certs/puppetmaster2.mozilla.com/client10.mozilla.com.crt
client_certs/puppetmaster2.mozilla.com/client11.mozilla.com.crt
client_certs/puppetmaster2.mozilla.com/revoke/

where there is a pending revocation on puppetmaster1. This client_certs directory is synchronized among all masters, and a crontask runs on each master that checks its revoke subdirectory for any pending revocations, and carries them out. After each such revocation, a new CRL is generated and placed in the certdir, where it is soon distributed again to all masters.

Commands

All of this is accomplished with the command we all love to hate, openssl. Here are some of the commands used for various parts of this process. This isn't a complete recipe because, honestly, you should struggle with this stuff a bit so that you understand it - it's easy to accidentally configure encryption to do absolutely nothing of value if you're not paying attention. I've tried to call out the processes that took me the longest to figure out, and the things I stumbled over, in hopes you can make the journey more quickly than I did.

NOTE: be careful to distinguish the puppetmaster's CA certificate from its leaf certificate, particularly in the Apache configurations.

Base CA

The base CA has a simple self-signed certificate. Its openssl.conf looks like this:

[ca]
default_ca = puppetagain-base-ca
 
[puppetagain-base-ca]
certificate = ./puppetagain-base-ca.crt
private_key = ./puppetagain-base-ca.key
database = ./inventory.txt
new_certs_dir = ./puppetagain-base-ca-certs
serial = ./serial
 
default_crl_days = 3650
default_days = 1825
default_md = sha1
 
policy = general_policy
x509_extensions = general_exts
 
[general_policy]
commonName = supplied
emailAddress = supplied
organizationName = supplied
organizationalUnitName = supplied
 
[general_exts]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:true
keyUsage = keyCertSign, cRLSign

Making a New Puppetmaster CA Certificate

$fqdn is the fqdn of the puppet master here. First, put a nice long password in ${fqdn}-ca.password. You'll never need to type it or even see it again.

$ openssl genrsa -des3 -out ${fqdn}-ca.key -passout file:${fqdn}-ca.password 2048
$ vi openssl.conf
[req]
prompt = no
distinguished_name = puppetmaster_ca_dn
[puppetmaster_ca_dn]
commonName = CA on ${fqdn}
emailAddress = youremail@domain.com
organizationalUnitName = Your OU
organizationName = Your Org

$ openssl req -config openssl.conf -new -key ${fqdn}-ca.key -out ${fqdn}-ca.csr -passin file:${fqdn}-ca.password
$ openssl req -text -in ${fqdn}-ca.csr

Check the request's contents in the dump, and copy it (${fqdn}-ca.csr) over to the machine hosting the base CA. There, run:

# openssl ca -config openssl.conf -in ${fqdn}-ca.csr -notext -out ${fqdn}-ca.crt -batch

and copy the resulting ${fqdn}-ca.crt back to the puppet master.

You'll need to set up an openssl.conf for signing *using* this new CA certificate:

[ca]
default_ca = server_ca

[server_ca]
certificate = ${fqdn}-ca.crt
private_key = ${fqdn}-ca.key
database = $PWD/inventory.txt
new_certs_dir = $PWD/certs
serial = $PWD/serial

default_crl_days = 7
default_days = 1825
default_md = sha1

policy = general_policy
x509_extensions = general_exts

[general_policy]
commonName = supplied

[general_exts]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:false
keyUsage = keyEncipherment, digitalSignature
extendedKeyUsage = serverAuth, clientAuth

# extensions used to sign the server cert (alternative name, in particular)
[servercert_exts]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:false
keyUsage = keyEncipherment, digitalSignature
extendedKeyUsage = serverAuth, clientAuth
# include the fqdn here to work around https://bugs.ruby-lang.org/issues/6493
subjectAltName = DNS:puppet,DNS:puppetmaster.fully.qualified.name

The salient points there are that the first two options in [server_ca] are the CA certificate and key you just generated; and the subjectAltName in the servercert_exts section (which will only be used to sign the leaf cert).

Note that the private key has a password, stored in ${ssldir}/ca/private/ca.pass. This path is what Puppet uses naturally, although we never allow puppet to touch it. This just adds a little more obfuscation, without any significant complexity.

You should generate a CRL right away, even though it will be empty:

openssl ca -config openssl.conf -gencrl -passin file:${ca_pass} -out "${certdir}/${fqdn}.crl"

Making a new Puppetmaster Leaf Certificate

cat <<EOF > openssl.conf
[req]
prompt = no
distinguished_name = servercert_dn

[servercert_dn]
commonName = ${fqdn}
EOF
openssl genrsa -des3 -out ${fqdn}.key-tmp -passout pass:temppass 2048
openssl rsa -in  ${fqdn}.key-tmp -passin pass:temppass -out ${fqdn}.key
rm ${fqdn}.key-tmp
openssl req -config openssl.conf -new -key ${fqdn}.key -out ${fqdn}.csr
openssl req -text -in ${fqdn}.csr
openssl ca -config ${ssldir}/ca/openssl.conf -in ${fqdn}.csr -notext -out ${fqdn}.crt -batch \
    -extensions servercert_exts -passin file:${ssldir}/ca/private/ca.pass

Here, the commonName is important, and it's important that it be the only component of the DN. I've assumed that the puppetmaster CA is in ${ssldir}/ca; you can put yours where you like.

Note that the resulting key has no passphrase, since it's used in an automated process.

Note, too, that this specifically calls out the servercert_exts, which is what triggers the subjectAltName to be present.

Signing a client cert

This looks suspiciously similar to signing the server cert, with the exception of not using the servercert_exts.

# make a key
openssl genrsa -des3 -out "/root/$host.key.pass" -passout pass:x

# strip its password
openssl rsa -in "/root/$host.key.pass" -out "/root/$host.key" -passin pass:x

# make a signing request
cat <<EOF > "/root/$host-openssl.conf"
[req]
prompt = no
distinguished_name = clientcert_dn

[clientcert_dn]
commonName = ${host}
EOF
openssl req -config "/root/$host-openssl.conf" -new -key "/root/${host}.key" -out "/root/${host}.csr"

# sign it
openssl ca -config "${ssldir}/ca/openssl.conf" -in "/root/${host}.csr" -notext -out "/root/${host}.crt" \
    -batch -passin "file:${ssldir}/ca/private/ca.pass"

Again, the commonName is important, and must be the only thing in the DN.

Revoking Certificates

This script runs in a crontask to revoke certificates and regenerate CRLs.

need_crl=false
shopt -s nullglob
cd "$client_certs_dir/$fqdn/revoke"
for crt in *.crt; do
    openssl ca -revoke $crt -config "$ssldir/ca/openssl.conf" \
            -keyfile "$ssldir/ca/ca_key.pem" -passin file:"$ssldir/ca/private/ca.pass" \
            -cert "$ssldir/ca/ca_crt.pem"
    rm "$crt"
    need_crl=true
done

# check the date on the CRL, regenerting if it's over a day old
if [ -n "$(find $crl -mtime +1 2>/dev/null)" ]; then
    need_crl=true
fi

# if we revoked anything or otherwise need to re-gen the CRL, do it
if $need_crl; then
    openssl ca -gencrl -config "$ssldir/ca/openssl.conf" \
            -keyfile "$ssldir/ca/ca_key.pem" -passin file:"$ssldir/ca/private/ca.pass" \
            -cert "$ssldir/ca/ca_crt.pem" -out "$certdir/$fqdn.crl"
fi

Gotchas and Tips

These were the points with lots of stars and uncouth language next to them in my journal. --Djmitche 21:58, 22 May 2012 (PDT)

• OpenSSL *must* trust the root of the certificate chain (the self-signed cert). It must have access to the intermediate certificates, but it doesn't matter if they're trusted. When I began this process, I signed the base CA with an internal Mozilla CA certificate. This would have worked, except that I was forced to trust *everything* signed by that certificate, which was too broad for my purposes.
• If you don't set ca = false on the master, it will do all sorts of nasty things, sending bogus certificates and CRLs to the client. I saw "The certificate retrieved from the master does not match the agent's private key".
• You must set certificate_revocation = false on the agent, or else the agent will try to download a CRL from the master, and then fail because that (single) CRL cannot completely cover the two CA certificates involved in the certification chain.
• In certificates used for SSL, the DN must contain only commonName=$hostname; in the 'openssl x509' output, you should see something like:

  Subject: CN=relabs08.mozilla.com

• note that in a certdir, certificates are hashed with an extension of '.0' (or .1 and so on if there are collisions), while CRLs are hashed with an extension ending of '.r0' (or .r1, ..). This is not obvious *anywhere* except in the output of a 'strace' of Apache (seriously).
• Apache only checks CRLs that it can find; a CA certificate without a corresponding CRL is assumed to have no revoked certificates. This can be helpful if you do not want to worry about expired CRLs, but could also be a security problem if you're not careful.
• Ruby's OpenSSL::SSL class ignores the distinguished name on a certificate if there are subjectAltNames defined. This matters for server certs, which must thus include both their fqdn and 'puppet' if they are to be usable at both names. See https://bugs.ruby-lang.org/issues/6493
• CRLs potentially expire very quickly. If you don't have plans in place to automatically regenerate CRLs, pick a very long expiration time for them (default_crl_days in openssl.conf).

The 'openssl verify' command is *very* useful for verifying that a set of certificates all line up. Sadly, it seems unable to work with a CRL directory, but it can take a bundle of CRLs appended to one another. If I recall correctly, order of those CRLs is important.

 openssl verify -purpose sslclient -crl_check_all -CApath /var/lib/puppet/ssl/certdir/ -CRLfile crl-bundle.pem relabs08.mozilla.com.crt

When debugging, start by using openssl s_client on the client, rather than running puppet. Puppet currently ignores the error codes from certificate validation, so it will give you very little data to debug with. I'm working on a patch for better reporting.

openssl s_client -connect puppet:8140 -verify 3 -CAfile certs/ca.pem 
   -cert certs/$client.pem -key private_keys/$client.pem

Look for two things: first, at the very bottom of the output, you'll see the overall result of the validation. Second, if you get your shell prompt back right away, then Apache is rejecting you.

You can add 'LogLevel Debug' to the Apache VirtualHost section to get *very* verbose logging of its SSL machinations.