PuppetDB: Failed to submit 'replace facts' command
Solution 1
I got it going, but can't say exactly what steps were necessary or not.
This issue started because authentication on several hosts was slow or hanging, and appeared to be related to domain controller/DNS cache issues. Removing domain mydomain.com
entry from /etc/resolv.conf
on the puppet master and agents solved the issue, but that created issues with existing puppet certs. I ran puppet cert clean --all
on the master to try and recreate all certs, but this did not play well with PuppetDB.
Solution
Clean out old certs on master:
puppet cert clean --all
Clean out old certs on all agents:
rm -rf /var/lib/puppet/ssl
Recreate PuppetDB keystores:
facter fqdn
is not available after removing domain foo.com
from /etc/resolv.conf
. This causes puppetdb-ssl-setup
to fail silently.
Edit /usr/sbin/puppetdb-ssl-setup
, add a piece of code to use just facter hostname
if facter fqdn
is empty:
# near line 10
fqdn=`facter fqdn`
# add this "if" section
if [ ! -n "$fqdn" ] ; then
fqdn=`facter hostname`
fi
Permissions fix:
chown -R puppetdb:puppetdb /etc/puppetdb/ssl
Update passwords in /etc/puppetdb/conf.d/jetty.ini with new keystore/truststore passcode (same pass), which you can get from:
cat /etc/puppetdb/ssl/puppetdb_keystore_pw.txt
Restart puppetdb
service puppetdb restart
Then go to each agent and request new certs and sign each on the master.
Solution 2
This also happens, when your memory settings for puppetdb are too low.
vim /etc/default/puppetdb
Edit the line
JAVA_ARGS="-Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"
should become
JAVA_ARGS="-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"
and restart puppetdb
sudo service puppetdb restart
Related videos on Youtube
Comments
-
Banjer almost 2 years
I recently revoked/cleaned a Puppet agent cert, and this seems to have negative effects in PuppetDB. I see a bug has been filed here with some instructions on fixing the issue. A user had a similar issue here, but none of this is working for me.
The server is running CentOS 6.2, Puppet 2.7.13, and Puppet DB 0.9. The error is:
root@harp:/etc/puppetdb/ssl> puppet agent --test err: Cached facts for harp failed: Failed to find facts from PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client info: Loading facts in /etc/puppet/modules/dns/lib/facter/datacenter.rb info: Caching facts for harp err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client err: Could not run Puppet configuration client: Could not retrieve local facts: Failed to submit 'replace facts' command for harp to PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed. This is often because the time is out of sync on the server or client
NTP is working properly from what I see and the datetime looks good. "harp" is actually the puppet master server, so there shouldn't be an issue with time between the agent and server here since they're the same.
Old certificate:
root@harp:/etc/puppetdb/ssl> puppet cert list --all + harp (DF:8F:65:36:58:4C:DE:66:2B:65:D1:E6:18:B7:F2:33)
Clean and generate new cert for agent:
root@harp:/etc/puppetdb/ssl> puppet cert clean harp notice: Revoked certificate with serial 18 notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/ca/signed/harp.pem' notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/certs/harp.pem' notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/certificate_requests/harp.pem' notice: Removing file Puppet::SSL::Key harp at '/var/lib/puppet/ssl/private_keys/harp.pem' root@harp:/etc/puppetdb/ssl> puppet agent --test info: Creating a new SSL key for harp warning: peer certificate won't be verified in this SSL session warning: peer certificate won't be verified in this SSL session info: Creating a new SSL certificate request for harp info: Certificate Request fingerprint (md5): 72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD warning: peer certificate won't be verified in this SSL session warning: peer certificate won't be verified in this SSL session warning: peer certificate won't be verified in this SSL session Exiting; no certificate found and waitforcert is disabled root@harp:/etc/puppetdb/ssl> puppet cert list harp (72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD) root@harp:/etc/puppetdb/ssl> puppet cert sign harp notice: Signed certificate request for harp notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/ca/requests/harp.pem' root@harp:/etc/puppetdb/ssl> puppet cert list --all + harp (4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79) root@harp:/etc/puppetdb/ssl> service puppetdb restart Stopping puppetdb: /etc/init.d/puppetdb: line 77: kill: (8623) - No such process [FAILED] Starting puppetdb: [ OK ]
OK then, restart again for good measure:
root@harp:/etc/puppetdb/ssl> service puppetdb restart Stopping puppetdb: [ OK ] Starting puppetdb: [ OK ]
Run the SSL configuration script
root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup cp: cannot stat `/var/lib/puppet/ssl/certs/harp.pem': No such file or directory root@harp:/etc/puppetdb/ssl> ls -la /var/lib/puppet/ssl/certs total 12 drwxr-xr-x 2 puppet root 4096 Jun 19 07:19 ./ drwxrwx--x 8 puppet root 4096 Apr 24 10:04 ../ -rw-r--r-- 1 puppet root 1854 Apr 24 10:04 ca.pem
OK then, try again for good measure:
root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup Certificate was added to keystore Usage: pkcs12 [options] where options are -export output PKCS12 file -chain add certificate chain -inkey file private key if not infile -certfile f add all certs in f -CApath arg - PEM format directory of CA's -CAfile arg - PEM format file of CA's -name "name" use name as friendly name -caname "nm" use nm as CA friendly name (can be used more than once). -in infile input filename ...snip... -CSP name Microsoft CSP name -LMK Add local machine keyset attribute to private key
It does not appear that the keystores in /etc/puppetdb/ssl have changed/regenerated. At this point, running
puppet agent --test
results in the same errors, and restarting puppet and puppetdb do not help.Keystore info:
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/keystore.jks Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry harp.mydomain.com, May 25, 2012, PrivateKeyEntry, Certificate fingerprint (MD5): 06:A8:D3:2A:70:F3:6D:34:62:91:45:22:8A:C4:A8:86 root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/truststore.jks Enter keystore password: Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry puppetdb ca, May 25, 2012, trustedCertEntry, Certificate fingerprint (MD5): 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88 root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp.mydomain.com ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88 err: Could not call fingerprint: Could not find a certificate or csr for harp.mydomain.com root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88 harp 4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79
How can I get the puppetdb keystore to actually regenerate? I tried deleting the files in /etc/puppetdb/ssl/, but no luck.
-
Daniël W. Crompton over 10 yearsFor me killing and restarting puppetdb was enough to solve this issue.