Puppet agent certificate verify failure

Question

I have a Puppet Master/Agent set up, and have successfully signed the certificate for the agent on the master. However, when I run puppet agent --test I get a failure that looks like this:

Warning: Unable to fetch my node definition, but the agent run will continue:  
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Info: Retrieving plugin  
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com] Could not retrieve file metadata for puppet://hostname.domain.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  
Warning: Not using cache on failed catalog   
Error: Could not retrieve catalog; skipping run  
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=hostname.domain.com]  

the hostname.domain.com is the master

How do I fix this? I've made sure that both clocks are at the correct time in the same time zone, I've deleted everything in the agent /var/lib/puppet/ssl directory and resigned, I don't know what else to do.

Solution 1

Re-create the entire client certificate setup. This has always fixed any cert issues we have experienced in the past. The following instruction assumes your agent's hostname is agenthost.hostname.com

On the client, delete all stored certs, including the CA:

find /var/lib/puppet/ssl -name '*.pem' -delete

On the master, delete any pending CSRs or old client certificates for this client:

find /var/lib/puppet/ssl -name agenthost.domain.com.pem -delete

Then, on the client, reconnect to the master and send a CSR:

puppet agent -t --waitforcert=60

and when it is waiting (if you have not set autosigning enabled) then on the master approve the CSR so a new client cert is sent back:

puppet cert sign agenthost.domain.com

This should make the agent re-download the puppet CA certificates, and re-apply for its own certificate.

We had to use this procedure in the past when we changed puppet servers and the CA certs changed, or when we rebuilt a host with the same hostname.

Make sure your agent knows its real fully-qualified hostname; use the 'hostname' command to ensure that it is what you expect it to be.

Solution 2

I have a similar problem. I have set up a vagrant environment with one puppetmaster and several clients. The problem is when I destroy and create the puppetmaster, clients detect the new puppetmaster as an impostor.

Deleting /etc/puppet/ssl on the client solves the problem.

Remember that your ssl configuration will be cached, so a restart of the puppet master is required, if you decide to also delete your /etc/puppet/ssl on that host:

sudo /etc/init.d/puppetmaster restart

Comments

[ravi yarlagadda]

Your master seems to be using a different certificate than your client trusts? Has the master's certificate changed?

[John Smith]

@ShaneMadden I do not think so...should I clean and revoke the masters and the clients certificates? I haven't messed with the masters certificates at all, but here's what the output of "puppet cert list --all" looks like: + "masterhost.domain.com" (SHA1) E1:F7:6A:21:CB:CD:xx:xx:xx:xx... + "agenthost.domain.com" (SHA256) 5A:D9:7B:96:0B:FF:E4:87:58:AF:00:xx:xx:xx:xx..

[ravi yarlagadda]

And that masterhost.domain.com is the same one as hostname.domain.com in your question, right? Let's try this, we'll see if the certificates verify manually; run openssl s_client -connect masterhost.domain.com:8140 -showcerts, and copy the certificate data (starts with -----BEGIN CERTIFICATE-----, include that line and the end certificate line) into a new file, then run openssl verify -CAfile /var/lib/puppet/ssl/certs/ca.pem /path/to/file/from/last/command, and see if it verifies.

[John Smith]

@ShaneMadden It seems like something is amiss.... When I ran the "-showcerts" command, it gave me two "begin" and "end" certificates, so I tried first adding one of those to a new file, and got this: /var/lib/puppet/ssl/ca/test: /CN=masterhost.domain.com error 7 at 0 depth lookup:certificate signature failure 22297:error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01:rsa_pk1.c:100: 22297:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:697: 22297:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:173:

[ravi yarlagadda]

That's.. strange. It sounds like it's sending the root certificate in addition to the server cert in the connection, so maybe just compare the contents of the second cert from -showcerts with the contents of /var/lib/puppet/ssl/certs/ca.pem - they should be identical?

[John Smith]

@ShaneMadden They are not! Ca.pem is different from the second output of the -showcerts command

[ravi yarlagadda]

Interesting! Can you get that second cert from -showcerts into a file, the compare openssl x509 -in /path/to/cert -noout -text between the two?

[Ivan Chau]

On Master, puppet cert clean ‘agentName’ works for me. RHEL Puppet Enterprise ssl location: /etc/puppetlabs/puppet/ssl. And I've synced the time between Master and Agent.

[Steve Shipway]

Puppet Enterprise, and the newer Puppet 4, store their certs under /etc/puppetlabs/puppet/ssl, as mentioned above. The /var/lib/puppet/ssl location is for the Puppet 2.x and 3.x community editions.

[Steve Shipway]

With puppet 5 or later, try puppetserver ca clean --certname $HOST on the Puppetserver, and puppet ssl clean on the client with the problem (use the client fqdn for $HOST)