ReleaseEngineering/PuppetAgain/Certificate Chaining
The Mozilla implementation of PuppetAgain has as one of its goals that any client can communicate freely with any master. Ordinarily, each puppetmaster has its own certificate authority, and a client which has been issued a certificate by one master will not be recognized by another master.
The solution is certificate chaining.
In this configuration, Puppet fits into a larger SSL certificate system. At Mozilla, this is specifically used to allow hosts to issue and validate all certificates without being directly connected. This supports resiliency, isolation of disclosure risk, and potentially (though not at Mozilla) integration into an enterprise-wide certification hierarchy.
Contents
Certificate Hierarchy
We have a hierarchy of certificate authorities that looks like this:
Root CA cert (Subject: CN=PuppetAgain Base CA/emailAddress=release@mozilla.com, OU=Release Engineering, O=Mozilla, Inc.)
|
+--Master CA cert (Subject: CN=CA on $fqdn)
| |
| +--Master cert (Subject: CN=$fqdn, OU=PuppetMasters)
| |
| +--Agent cert (Subject: CN=$fqdn)
| |
| +--Agent cert (Subject: CN=$fqdn)
| :
| .
|
+-- Master CA cert (Subject: CN=CA on $fqdn)
: |
. +--Master cert (Subject: CN=$fqdn, OU=PuppetMasters)
|
+--Agent cert (Subject: CN=$fqdn)
|
+--..
:
.
Notes:
- Here and throughout these docs, the terms "root" and "base" CA are used interchangeably. "Root" is preferred.
- Master and Agent certs are nearly identical, except that one has an EKU allowing use as SSL clients, and the other only allows SSL server. The "OU=PuppetMasters" is important so that each puppetmaster can have a master and agent cert with the same fqdn, but the certificate subjects are different.
Master Initialization
When a master is initialized, it is provided with a puppetmaster CA certificate and corresponding key. This certificate is signed by the base CA. The puppet manifests for masters immediately create a Master cert, suitable for authenticating an SSL connection, signed by this Master CA cert.
Certificate Issuance
When a client is issued a new certificate, after the requisite authentication has been performed (see Puppetization Process), a new certificate and key are generated and signed by the CA on the puppetmaster doing the issuing. Any puppetmaster can issue a certificate. This certificate and key, along with the root CA certificate, are returned to the client.
The process looks something like this: (although see the puppet manifests themselves to see how this works for real)
# make a key
openssl genrsa -des3 -out "/root/${host}.key.pass" -passout pass:x
# strip its password
openssl rsa -in "/root/${host}.key.pass" -out "/root/${host}.key" -passin pass:x
# make a signing request
cat <<EOF > "/root/${host}-openssl.conf"
[req]
prompt = no
distinguished_name = clientcert_dn
[clientcert_dn]
commonName = ${host}
EOF
openssl req -config "/root/${host}-openssl.conf" -new -key "/root/${host}.key" -out "/root/${host}.csr"
# sign it (this openssl.conf points to the local Master CA cert)
openssl ca -config "${master_ssldir}/ca/openssl.conf" -in "/root/${host}.csr" -notext -out "/root/${host}.crt" \
-batch -passin "file:${master_ssldir}/ca/private/ca.pass"
In particular, the `commonName` is the fqdn of the host. Agents are configured with the Root CA cert, the Agent cert, and the private key for the Agent cert.
Master CA certificates are generated by hand, using a process something like this:
(put a nice long password in ${fqdn}-ca.password)
$ openssl genrsa -des3 -out ${fqdn}-ca.key -passout file:${fqdn}-ca.password 2048
$ vi openssl.conf
[req]
prompt = no
distinguished_name = puppetmaster_ca_dn
[puppetmaster_ca_dn]
commonName = CA on ${fqdn}
emailAddress = release@mozilla.com
organizationalUnitName = Release Engineering
organizationName = Mozilla, Inc.
$ openssl req -config openssl.conf -new -key ${fqdn}-ca.key -out ${fqdn}-ca.csr -passin file:${fqdn}-ca.password
$ openssl req -text -in ${fqdn}-ca.csr
The resulting CSR is then signed by the Root CA cert on an isolated system.
Validation
The relevant Apache configuration looks like this:
SSLCertificateFile /var/lib/puppet/ssl-master/certs/puppet.pem
SSLCertificateKeyFile /var/lib/puppet/ssl-master/private_keys/puppet.pem
# include the intermediate cert (this host's CA)
SSLCertificateChainFile /var/lib/puppet/ssl-master/ca/ca_crt.pem
# certs and CRLs for all CAs (the base, plus the CA for each puppet
# master) are stored in the same dir. This dir is synchronized with csync2.
SSLCACertificatePath /var/lib/puppet/ssl-master/certdir
SSLCARevocationPath /var/lib/puppet/ssl-master/certdir
# verification is optional; the puppet rails app looks at SSL_CLIENT_VERIFY
# and auth.conf to decide what requires verification
SSLVerifyClient optional
# puppetagain cert chains have two CA's, so verify twice the depth
SSLVerifyDepth 2
SSLOptions +StdEnvVars
# The following client headers allow the same configuration to work with Pound.
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
The certdir contains all puppetmaster CA certs and their CRL's, as well as the base CA cert and its CRL. We synchronize this directory among all puppetmasters using csync2. The certificates and CRLs are hashed using the following script:
for i in *.crl; do h=`openssl crl -hash -noout -in $i` fn=$h.r0 [ ! -f $fn ] && ln -s $i $fn done for i in *.crt; do h=`openssl x509 -hash -noout -in $i` fn=$h.0 [ ! -f $fn ] && ln -s $i $fn done
Validation of Agents
Apache takes care of validating agents. The agents provide their Agent cert, and Apache searches the certdir for parent certificates until it reaches a valid self-signed cert. Concretely, in this case it finds the Master CA cert that signed the Agent cert in certdir, and then finds the Root CA cert there.
Validation of Masters
Agents are configured with the Root CA cert. Apache's SSLCertificateChainFile directive supplies the intermediate Master CA cert, and the SSLCertificateFile provides the Master cert. These three certificates constitute the entire chain required for validation of the master by the agent.
CRLs
Certificate revocation lists (CRLs) are crucial to puppet's operation, as certificates are often re-issued to clients, e.g., when they are re-imaged.
All of the CA's described above have corresponding CRLs, and these are made available to Apache with the SSLCARevocationPath directive, as seen above.
Validation
Apache consults its SSLCARevocationPath to check the corresponding CRLs for all CA certificates. This directory is kept up to date by each master, and synchronized between masters using a cronjob.
Unfortunately, until 14550 is fixed, the puppet agent cannot validate a certificate chain's CRLs, so CRL checking is disabled on the agent:
certificate_revocation = false
This means that agents may continue to trust a compromised Master cert or Master CA cert.
Revocation
When it comes time to revoke a certificate, things can be a bit complicated. Revocation of a compromised cert requires access to the CA certificate and key, but that key only exists on the master that created the compromised cert it. So a method is required to indicate that a particular certificate should be revoked. This is accomplished with a number of directories in the git repository shared between the masters. The relevant pieces look like this:
agent-certs/relabs02.build.mtv1.mozilla.com/ agent-certs/relabs03.build.mtv1.mozilla.com/ agent-certs/relabs03.build.mtv1.mozilla.com/agent1.build.mtv1.mozilla.com.crt agent-certs/relabs03.build.mtv1.mozilla.com/agent2.build.mtv1.mozilla.com.crt revocation-requests/relabs02.build.mtv1.mozilla.com/ revocation-requests/relabs02.build.mtv1.mozilla.com/agent8.build.mtv1.mozilla.com-for-relabs03.build.mtv1.mozilla.com.crt revocation-requests/relabs03.build.mtv1.mozilla.com/
Active agent certs are kept in agent-certs, under a directory named for the master that issued the cert. When any master wants to revoke a certificate, it moves it ('git mv') to a subdirectory of revocation-requests again named after the issuing master. The filename is set up to be unique even if two masters request a revocation of the same certificate.
A crontask runs on each master that checks its revocation requests for any pending revocations, and carries them out. After each such revocation, a new CRL is generated and placed in the certdir, where it is soon distributed again to all masters.
Puppet Configuration
On the master, you'll need to convince it not to try to do fancy CA-like things:
ca = false
Command Reference
All of this is accomplished with the command we all love to hate, openssl. Here are some of the commands used for various parts of this process. This isn't a complete recipe because, honestly, you should struggle with this stuff a bit so that you understand it - it's easy to accidentally configure encryption to do absolutely nothing of value if you're not paying attention. I've tried to call out the processes that took me the longest to figure out, and the things I stumbled over, in hopes you can make the journey more quickly than I did.
NOTE: be careful to distinguish the puppetmaster's CA certificate from its leaf certificate, particularly in the Apache configurations.
Root CA
The root CA has a simple self-signed certificate. This is the keys to the kingdom, so be careful with it. Put it on a well-protected system, isolated from your puppet environment, and protect the passphrase carefully.
You should also understand the difference between a certificate, a key, and a CRL. There are plenty of good summaries out there on the 'net.
Put the following in openssl.conf:
[ca] default_ca = puppetagain-base-ca [puppetagain-base-ca] certificate = ./puppetagain-base-ca.crt private_key = ./puppetagain-base-ca.key database = ./inventory.txt new_certs_dir = ./puppetagain-base-ca-certs serial = ./serial default_crl_days = 3650 default_days = 1825 default_md = sha1 policy = general_policy x509_extensions = general_exts [general_policy] commonName = supplied emailAddress = supplied organizationName = supplied organizationalUnitName = supplied [general_exts] authorityKeyIdentifier=keyid,issuer:always basicConstraints = critical,CA:true keyUsage = keyCertSign, cRLSign
then touch inventory.txt and echo 0001 > serial.
Set up a new self-signed CA cert with:
openssl req -new -newkey rsa -days 3650 -x509 -subj "/CN=PuppetAgain Base CA, OU=Release Engineering, O=Mozilla, Inc." -keyout puppetagain-base-ca.key -out puppetagain-base-ca.crt
adjusting the subject appropriately for your environment. The subject doesn't particularly matter, but using the above will risk it being confused with moco's certificate.
Generate a CRL with
openssl ca -config openssl.conf -gencrl -out puppetagain-base-ca.crl
You now have a *.key (private key - keep this secret!), *.crt (certificate), and *.crl (CRL) file for your root CA.
Note that the file contents are short blobs encoded in a text format. You can easily copy-and-paste them, if -- as is wise -- your CA host is strictly isolated from your production systems.
Making a New Puppetmaster CA Certificate
You should already know what a key, certificate, CRL, and CSR are.
The idea here is to make a CA certificate (one that can sign other certificates) that is signed by the root CA.
The following commands will make a new key (master.key) and a corresponding CSR. Note that the instructions you get from puppet when you're setting this up will contain explicit paths, so it will be easier to copy/paste there.
openssl genrsa -out ${master_ca_key} 2048
openssl req -new -subj "/CN=CA on ${fqdn}" -key ${master_ca_key} -out master-ca.csr
openssl req -text -in master-ca.csr
Check that the CSR has the expected fields (check the dates, etc.), then copy/paste it into a temporary file (say, master.csr) on the host where your root CA is set up. There, run
openssl ca -config openssl.conf -in master.csr
This is using the root CA to sign the master CA's certificate. Check that the CSR values match what you specified above, and answer the prompts. You'll end up with a new certificate, which you can copy and paste back onto the puppetmaster (the puppet message will tell you where to put it).
Puppetmaster CA Setup
You don't need to know this if you're using PuppetAgain, because the setup scripts do it for you (and slightly differently), but for those wondering how Certificate Chaining works in general:
$ openssl genrsa -des3 -out ${fqdn}-ca.key -passout file:${fqdn}-ca.password 2048
$ vi openssl.conf
[req]
prompt = no
distinguished_name = puppetmaster_ca_dn
[puppetmaster_ca_dn]
commonName = CA on ${fqdn}
emailAddress = youremail@domain.com
organizationalUnitName = Your OU
organizationName = Your Org
$ openssl req -config openssl.conf -new -key ${fqdn}-ca.key -out ${fqdn}-ca.csr -passin file:${fqdn}-ca.password
$ openssl req -text -in ${fqdn}-ca.csr
Check the request's contents in the dump, and copy it (${fqdn}-ca.csr) over to the machine hosting the base CA. There, run:
# openssl ca -config openssl.conf -in ${fqdn}-ca.csr -notext -out ${fqdn}-ca.crt -batch
and copy the resulting ${fqdn}-ca.crt back to the puppet master.
You'll need to set up an openssl.conf for signing *using* this new CA certificate:
[ca]
default_ca = server_ca
[server_ca]
certificate = ${fqdn}-ca.crt
private_key = ${fqdn}-ca.key
database = $PWD/inventory.txt
new_certs_dir = $PWD/certs
serial = $PWD/serial
default_crl_days = 7
default_days = 1825
default_md = sha1
policy = general_policy
x509_extensions = general_exts
[general_policy]
commonName = supplied
[general_exts]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:false
keyUsage = keyEncipherment, digitalSignature
extendedKeyUsage = serverAuth, clientAuth
# extensions used to sign the server cert (alternative name, in particular)
[servercert_exts]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints = critical,CA:false
keyUsage = keyEncipherment, digitalSignature
extendedKeyUsage = serverAuth, clientAuth
# include the fqdn here to work around https://bugs.ruby-lang.org/issues/6493
subjectAltName = DNS:puppet,DNS:puppetmaster.fully.qualified.name
The salient points there are that the first two options in [server_ca] are the CA certificate and key you just generated; and the subjectAltName in the servercert_exts section (which will only be used to sign the leaf cert).
Note that the private key has a password, stored in ${ssldir}/ca/private/ca.pass. This path is what Puppet uses naturally, although we never allow puppet to touch it. This just adds a little more obfuscation, without any significant complexity.
You should generate a CRL right away, even though it will be empty:
openssl ca -config openssl.conf -gencrl -passin file:${ca_pass} -out "${certdir}/${fqdn}.crl"
Making a new Puppetmaster Leaf Certificate
cat <<EOF > openssl.conf
[req]
prompt = no
distinguished_name = servercert_dn
[servercert_dn]
commonName = ${fqdn}
EOF
openssl genrsa -des3 -out ${fqdn}.key-tmp -passout pass:temppass 2048
openssl rsa -in ${fqdn}.key-tmp -passin pass:temppass -out ${fqdn}.key
rm ${fqdn}.key-tmp
openssl req -config openssl.conf -new -key ${fqdn}.key -out ${fqdn}.csr
openssl req -text -in ${fqdn}.csr
openssl ca -config ${ssldir}/ca/openssl.conf -in ${fqdn}.csr -notext -out ${fqdn}.crt -batch \
-extensions servercert_exts -passin file:${ssldir}/ca/private/ca.pass
Here, the commonName is important, and it's important that it be the only component of the DN. I've assumed that the puppetmaster CA is in ${ssldir}/ca; you can put yours where you like.
Note that the resulting key has no passphrase, since it's used in an automated process.
Note, too, that this specifically calls out the servercert_exts, which is what triggers the subjectAltName to be present.
Signing a client cert
This looks suspiciously similar to signing the server cert, with the exception of not using the servercert_exts.
# make a key
openssl genrsa -des3 -out "/root/$host.key.pass" -passout pass:x
# strip its password
openssl rsa -in "/root/$host.key.pass" -out "/root/$host.key" -passin pass:x
# make a signing request
cat <<EOF > "/root/$host-openssl.conf"
[req]
prompt = no
distinguished_name = clientcert_dn
[clientcert_dn]
commonName = ${host}
EOF
openssl req -config "/root/$host-openssl.conf" -new -key "/root/${host}.key" -out "/root/${host}.csr"
# sign it
openssl ca -config "${ssldir}/ca/openssl.conf" -in "/root/${host}.csr" -notext -out "/root/${host}.crt" \
-batch -passin "file:${ssldir}/ca/private/ca.pass"
Again, the commonName is important, and must be the only thing in the DN.
Revoking Certificates
This script runs in a crontask to revoke certificates and regenerate CRLs.
need_crl=false
shopt -s nullglob
cd "$client_certs_dir/$fqdn/revoke"
for crt in *.crt; do
openssl ca -revoke $crt -config "$ssldir/ca/openssl.conf" \
-keyfile "$ssldir/ca/ca_key.pem" -passin file:"$ssldir/ca/private/ca.pass" \
-cert "$ssldir/ca/ca_crt.pem"
rm "$crt"
need_crl=true
done
# check the date on the CRL, regenerting if it's over a day old
if [ -n "$(find $crl -mtime +1 2>/dev/null)" ]; then
need_crl=true
fi
# if we revoked anything or otherwise need to re-gen the CRL, do it
if $need_crl; then
openssl ca -gencrl -config "$ssldir/ca/openssl.conf" \
-keyfile "$ssldir/ca/ca_key.pem" -passin file:"$ssldir/ca/private/ca.pass" \
-cert "$ssldir/ca/ca_crt.pem" -out "$certdir/$fqdn.crl"
fi
Gotchas and Tips
These were the points with lots of stars and uncouth language next to them in my journal. --Djmitche 21:58, 22 May 2012 (PDT)
- OpenSSL *must* trust the root of the certificate chain (the self-signed cert). It must have access to the intermediate certificates, but it doesn't matter if they're trusted. When I began this process, I signed the base CA with an internal Mozilla CA certificate. This would have worked, except that I was forced to trust *everything* signed by that certificate, which was too broad for my purposes.
- If you don't set ca = false on the master, it will do all sorts of nasty things, sending bogus certificates and CRLs to the client. I saw "The certificate retrieved from the master does not match the agent's private key".
- You must set certificate_revocation = false on the agent, or else the agent will try to download a CRL from the master, and then fail because that (single) CRL cannot completely cover the two CA certificates involved in the certification chain.
- In certificates used for SSL, the DN must contain only commonName=$hostname; in the 'openssl x509' output, you should see something like:
Subject: CN=relabs08.mozilla.com
- note that in a certdir, certificates are hashed with an extension of '.0' (or .1 and so on if there are collisions), while CRLs are hashed with an extension ending of '.r0' (or .r1, ..). This is not obvious *anywhere* except in the output of a 'strace' of Apache (seriously).
- Apache only checks CRLs that it can find; a CA certificate without a corresponding CRL is assumed to have no revoked certificates. This can be helpful if you do not want to worry about expired CRLs, but could also be a security problem if you're not careful.
- Ruby's OpenSSL::SSL class ignores the distinguished name on a certificate if there are subjectAltNames defined. This matters for server certs, which must thus include both their fqdn and 'puppet' in the subjectAltNames option if they are to be usable at both names. See https://bugs.ruby-lang.org/issues/6493
- CRLs potentially expire very quickly. If you don't have plans in place to automatically regenerate CRLs, pick a very long expiration time for them (default_crl_days in openssl.conf).
The 'openssl verify' command is *very* useful for verifying that a set of certificates all line up. Sadly, it seems unable to work with a CRL directory, but it can take a bundle of CRLs appended to one another. If I recall correctly, order of those CRLs is important.
openssl verify -purpose sslclient -crl_check_all -CApath /var/lib/puppet/ssl/certdir/ -CRLfile crl-bundle.pem relabs08.mozilla.com.crt
When debugging, start by using openssl s_client on the client, rather than running puppet. Puppet currently ignores the error codes from certificate validation, so it will give you very little data to debug with. I'm working on a patch for better reporting.
openssl s_client -connect puppet:8140 -verify 3 -CAfile certs/ca.pem -cert certs/$client.pem -key private_keys/$client.pem
Look for two things: first, at the very bottom of the output, you'll see the overall result of the validation. Second, if you get your shell prompt back right away, then Apache is rejecting you.
You can add 'LogLevel Debug' to the Apache VirtualHost section to get *very* verbose logging of its SSL machinations.
