Puppet: certificate verify failed

Question

Due to a screw up, I have to regenerate client & server certificates.

As far as I know, the master certificates are automatically generated.

So I generated keys on the client:

MASTER # puppet cert clean --all
Notice: Revoked certificate with serial 2
Notice: Revoked certificate with serial 6
Notice: Removing file Puppet::SSL::Certificate puppet.x.com at '/var/lib/puppet/ssl/ca/signed/puppet.x.com.pem'
Notice: Removing file Puppet::SSL::Certificate puppet.x.com at '/var/lib/puppet/ssl/certs/puppet.x.com.pem'
Notice: Removing file Puppet::SSL::Key puppet.x.com at '/var/lib/puppet/ssl/private_keys/puppet.x.com.pem'
Notice: Removing file Puppet::SSL::Certificate efikamx-9ba3ab.x.com at '/var/lib/puppet/ssl/ca/signed/efikamx-9ba3ab.x.com.pem'
Notice: Removing file Puppet::SSL::Certificate efikamx-9ba3ab.x.com at '/var/lib/puppet/ssl/certs/efikamx-9ba3ab.x.com.pem'

puppet agent --no-daemonize  --onetime --verbose --waitforcert 60 
notice: Did not receive certificate
info: Caching certificate for efikamx-561a37.botnet.corp.flatturtle.com
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
notice: Using cached catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

MASTER # puppet cert sign --all
Notice: Signed certificate request for efikamx-9ba3ab.x.com
Notice: Removing file Puppet::SSL::CertificateRequest efikamx-9ba3ab.x.com at '/var/lib/puppet/ssl/ca/requests/efikamx-9ba3ab.x.com.pem'

CLIENT # puppet agent -t
info: Caching certificate for efikamx-9ba3ab.x.com
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

Before you ask, yes, ntp is running and both clients have the correct time.

What's the correct way to purge all the certificates on both ends and correctly regenerate everything?

I've ran:

find /var/lib/puppet -type f -print0 |xargs -0r rm

and

rm -rf /var/lib/puppet/ssl/*

on the client, but that did not help.

This is a mixture of Puppet 2 and Puppet 3 by the way.

Solution

Apparently the problem lied in the fact that Apache was still running (and thus having a puppet master spawned via Passenger).

MASTER /etc/apache2/sites-enabled # /etc/init.d/apache2 stop
[ ok ] Stopping web server: apache2 ... waiting .
MASTER /etc/apache2/sites-enabled # puppet cert clean --all
Notice: Revoked certificate with serial 2
Notice: Removing file Puppet::SSL::Certificate puppet.x at '/var/lib/puppet/ssl/ca/signed/puppet.x.pem'
Notice: Removing file Puppet::SSL::Certificate puppet.x at '/var/lib/puppet/ssl/certs/puppet.x.pem'
Notice: Removing file Puppet::SSL::Key puppet.x at '/var/lib/puppet/ssl/private_keys/puppet.x.pem'
MASTER /etc/apache2/sites-enabled # puppet master --no-daemonize --verbose
Info: Creating a new SSL key for puppet.x
Info: Creating a new SSL certificate request for puppet.x
Info: Certificate Request fingerprint (SHA256): DB:8C:2D:71:54:C4:B7:03:79:38:E2:26:94:51:12:89:6F:E0:24:AC:F2:16:C0:5A:7A:B6:7D:4F:DD:6C:98:0D
Notice: puppet.x has a waiting certificate request
Notice: Signed certificate request for puppet.x
Notice: Removing file Puppet::SSL::CertificateRequest puppet.x at '/var/lib/puppet/ssl/ca/requests/puppet.x.pem'
Notice: Removing file Puppet::SSL::CertificateRequest puppet.x at '/var/lib/puppet/ssl/certificate_requests/puppet.x.pem'
Notice: Starting Puppet master version 3.1.1
^CNotice: Caught INT; calling stop
MASTER /etc/apache2/sites-enabled # /etc/init.d/apache2 restart
[ ok ] Restarting web server: apache2.
MASTER /etc/apache2/sites-enabled # puppet cert sign --all
Notice: Signed certificate request for efikamx-561a37.x
Notice: Removing file Puppet::SSL::CertificateRequest efikamx-561a37.x at '/var/lib/puppet/ssl/ca/requests/efikamx-561a37.x.pem'

And now I can correctly generate and sign the keys on the client:

CLIENT ~ # rm -rf /var/lib/puppet/ssl/*
CLIENT ~ # puppet agent -t
info: Creating a new SSL key for efikamx-9ba3ab.x.com
info: Caching certificate for ca
info: Creating a new SSL certificate request for efikamx-9ba3ab.x.com
info: Certificate Request fingerprint (md5): 8C:9E:6E:95:B8:70:B9:A2:98:CB:A5:87:BC:66:33:A4
Exiting; no certificate found and waitforcert is disabled
CLIENT ~ # puppet agent --no-daemonize  --onetime --verbose --waitforcert 60
info: Caching certificate for efikamx-9ba3ab.x.com
info: Caching certificate_revocation_list for ca
info: Caching catalog for efikamx-9ba3ab.x.com
info: Applying configuration version '1373327419'
notice: /Stage[essential]/Efikamx-repository/File[/etc/apt/sources.list.d/multistrap-stable.list]/content: content changed '{md5}fbba0743add1cb9e54f7484b2c7a1f59' to '{md5}5941829a1b3a18b02f5bd6367e36e635'
[...]

Comments

[Zoredache]

When you say mixture of puppet 2 & 3, which one is v3? The docs strongly suggest that the puppetmaster should always be newer than any clients.

[Tuinslak]

The master is version 3.1.1; the clients are either 2.7.x or 3.x (right now, I'm testing with two 2.7 clients)