Puppet's automatically generated certificates failing

puppet openssl puppetmaster

10,168

Solution 1

I've seen this before when something has gone weird with the cert signing process. The easiest way to fix this is to regen the client certs.

revoke the cert: puppetca revoke client.example.com
run puppetca clean on the client
stop puppet on the client
remove everything in the /var/lib/puppet/ssl directory
start puppet on the client
do an initial run I tend to use puppet agent --test
sign the new certificate on the server
test the client

Solution 2

I just hit a similar issue on CentOS 6.3. The problem was dat/etime out of sync. I fixed that and all worked just fine.

Solution 3

Make sure your not out of disk space. Puppet will not warn of this it will just create certificates that are of 0 length.

Solution 4

Check the date/time on the client system. If it is significantly out of sync, you will get SSL authentication errors.

View more solutions

10,168

gparent

Updated on September 18, 2022

Comments

gparent almost 2 years

I am running a default configuration of Puppet on Debian Squeeze 6.0.4.

The server's FQDN is master.example.com. The client's FQDN is client.example.com.

I am able to contact the puppet master and send a CSR. I sign it using puppetca -sa but the client will still not connect.

The two machines have accurate date and time.

This is what appears in /var/log/syslog:

Apr  3 17:03:52 localhost puppet-agent[18653]: Reopening log files
Apr  3 17:03:52 localhost puppet-agent[18653]: Starting Puppet client version 2.6.2
Apr  3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Apr  3 17:03:53 localhost puppet-agent[18653]: Using cached catalog
Apr  3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog; skipping run

Here is some interesting output:

OpenSSL client test:

client:~# openssl s_client -host master.example.com -port 8140 -cert /var/lib/puppet/ssl/certs/client.example.com.pem -key /var/lib/puppet/ssl/private_keys/client.example.com.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem
CONNECTED(00000003)
depth=1 /CN=Puppet CA: master.example.com
verify return:1
depth=0 /CN=master.example.com
verify error:num=7:certificate signature failure
verify return:1
depth=0 /CN=master.example.com
verify return:1
18509:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error:s3_pkt.c:1102:SSL alert number 51
18509:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:
client:~#

master's certificate:

root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/master.example.com.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 2 (0x2)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:28 2012 GMT
        Not After : Apr  2 20:01:28 2017 GMT
    Subject: CN=master.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:a9:c1:f9:4c:cd:0f:68:84:7b:f4:93:16:20:44:
                7a:2b:05:8e:57:31:05:8e:9c:c8:08:68:73:71:39:
                c1:86:6a:59:93:6e:53:aa:43:11:83:5b:2d:8c:7d:
                54:05:65:c1:e1:0e:94:4a:f0:86:58:c3:3d:4f:f3:
                7d:bd:8e:29:58:a6:36:f4:3e:b2:61:ec:53:b5:38:
                8e:84:ac:5f:a3:e3:8c:39:bd:cf:4f:3c:ff:a9:65:
                09:66:3c:ba:10:14:69:d5:07:57:06:28:02:37:be:
                03:82:fb:90:8b:7d:b3:a5:33:7b:9b:3a:42:51:12:
                b3:ac:dd:d5:58:69:a9:8a:ed
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:FALSE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
    7b:2c:4f:c2:76:38:ab:03:7f:c6:54:d9:78:1d:ab:6c:45:ab:
    47:02:c7:fd:45:4e:ab:b5:b6:d9:a7:df:44:72:55:0c:a5:d0:
    86:58:14:ae:5f:6f:ea:87:4d:78:e4:39:4d:20:7e:3d:6d:e9:
    e2:5e:d7:c9:3c:27:43:a4:29:44:85:a1:63:df:2f:55:a9:6a:
    72:46:d8:fb:c7:cc:ca:43:e7:e1:2c:fe:55:2a:0d:17:76:d4:
    e5:49:8b:85:9f:fa:0e:f6:cc:e8:28:3e:8b:47:b0:e1:02:f0:
    3d:73:3e:99:65:3b:91:32:c5:ce:e4:86:21:b2:e0:b4:15:b5:
    22:63
root@master:/etc/puppet#

CA's certificate:

root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/ca.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:05 2012 GMT
        Not After : Apr  2 20:01:05 2017 GMT
    Subject: CN=Puppet CA: master.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:b5:2c:3e:26:a3:ae:43:b8:ed:1e:ef:4d:a1:1e:
                82:77:78:c2:98:3f:e2:e0:05:57:f0:8d:80:09:36:
                62:be:6c:1a:21:43:59:1d:e9:b9:4d:e0:9c:fa:09:
                aa:12:a1:82:58:fc:47:31:ed:ad:ad:73:01:26:97:
                ef:d2:d6:41:6b:85:3b:af:70:00:b9:63:e9:1b:c3:
                ce:57:6d:95:0e:a6:d2:64:bd:1f:2c:1f:5c:26:8e:
                02:fd:d3:28:9e:e9:8f:bc:46:bb:dd:25:db:39:57:
                81:ed:e5:c8:1f:3d:ca:39:cf:e7:f3:63:75:f6:15:
                1f:d4:71:56:ed:84:50:fb:5d
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:TRUE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Certificate Sign, CRL Sign
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
Signature Algorithm: sha1WithRSAEncryption
    1d:cd:c6:65:32:42:a5:01:62:46:87:10:da:74:7e:8b:c8:c9:
    86:32:9e:c2:2e:c1:fd:00:79:f0:ef:d8:73:dd:7e:1b:1a:3f:
    cc:64:da:a3:38:ad:49:4e:c8:4d:e3:09:ba:bc:66:f2:6f:63:
    9a:48:19:2d:27:5b:1d:2a:69:bf:4f:f4:e0:67:5e:66:84:30:
    e5:85:f4:49:6e:d0:92:ae:66:77:50:cf:45:c0:29:b2:64:87:
    12:09:d3:10:4d:91:b6:f3:63:c4:26:b3:fa:94:2b:96:18:1f:
    9b:a9:53:74:de:9c:73:a4:3a:8d:bf:fa:9c:c0:42:9d:78:49:
    4d:70
root@master:/etc/puppet#

Client's certificate:

client:~# openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/client.example.com.pem
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 3 (0x3)
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: CN=Puppet CA: master.example.com
    Validity
        Not Before: Apr  2 20:01:36 2012 GMT
        Not After : Apr  2 20:01:36 2017 GMT
    Subject: CN=client.example.com
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
        RSA Public Key: (1024 bit)
            Modulus (1024 bit):
                00:ae:88:6d:9b:e3:b1:fc:47:07:d6:bf:ea:53:d1:
                14:14:9b:35:e6:70:43:e0:58:35:76:ac:c5:9d:86:
                02:fd:77:28:fc:93:34:65:9d:dd:0b:ea:21:14:4d:
                8a:95:2e:28:c9:a5:8d:a2:2c:0e:1c:a0:4c:fa:03:
                e5:aa:d3:97:98:05:59:3c:82:a9:7c:0e:e9:df:fd:
                48:81:dc:33:dc:88:e9:09:e4:19:d6:e4:7b:92:33:
                31:73:e4:f2:9c:42:75:b2:e1:9f:d9:49:8c:a7:eb:
                fa:7d:cb:62:22:90:1c:37:3a:40:95:a7:a0:3b:ad:
                8e:12:7c:6e:ad:04:94:ed:47
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:FALSE
        Netscape Comment:
            Puppet Ruby/OpenSSL Internal Certificate
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Subject Key Identifier:
            8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
        X509v3 Extended Key Usage: critical
            TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
    33:1f:ec:3c:91:5a:eb:c6:03:5f:a1:58:60:c3:41:ed:1f:fe:
    cb:b2:40:11:63:4d:ba:18:8a:8b:62:ba:ab:61:f5:a0:6c:0e:
    8a:20:56:7b:10:a1:f9:1d:51:49:af:70:3a:05:f9:27:4a:25:
    d4:e6:88:26:f7:26:e0:20:30:2a:20:1d:c4:d3:26:f1:99:cf:
    47:2e:73:90:bd:9c:88:bf:67:9e:dd:7c:0e:3a:86:6b:0b:8d:
    39:0f:db:66:c0:b6:20:c3:34:84:0e:d8:3b:fc:1c:a8:6c:6c:
    b1:19:76:65:e6:22:3c:bf:ff:1c:74:bb:62:a0:46:02:95:fa:
    83:41
client:~#

EDIT:

Someone suggested redoing the certificates. I have done this several times and it does not fix the issue.

gparent about 12 years

See question for this.
gparent about 12 years

I have done this several times. I am looking for a way to diagnose more specifically what is going on.
Kyle Smith about 12 years

Ah, thanks for updating the question to include this information.
gparent about 12 years

It was actually in the very first revision, but no problem.
gparent over 11 years

I'll accept your answer even if it didn't solve the issue because it's probably the most logical one for anyone else who gets problems.
Aaron R. almost 10 years

Thanks @RobinBowes, this was exactly my issue. It happened to me because I was rebuilding a server
Greg Bray about 4 years

Fun fact... steps 3 and 5 are REALLY important :-D If you happen to forget to restart the puppet agent service then manual puppet agent --test invocations will work fine and you will see the previous agent process print the correct Certificate fingerprint: ... value in syslog on interval runs (or when forced using pkill -SIGUSR1 puppet). But those runs will fail with Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch errors, likely because the private key resident in memory doesn't match the new public cert. Thanks Zypher!