PuppetDB: Connection refused

Question

I'm trying to get PuppetDB to run on my puppetmaster machine. The problem is that I always get the following message when doing an agent run:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for handsoff.dc0.testing.de to PuppetDB at handsoff:8081: Connection refused - connect(2)

The puppetmaster, puppetdb and database are all running on the same host "handsoff".

This is my /etc/puppet/puppetdb.conf

[main]
server = handsoff
port = 8081

My /etc/puppetdb/jetty.ini

[jetty]
# Hostname or IP address to listen for clear-text HTTP.  Default is localhost
# host = <host>

# Port to listen on for clear-text HTTP.
port = 8080


# The following are SSL specific settings. They can be configured
# automatically with the tool `puppetdb ssl-setup`, which is normally
# ran during package installation.

# The host or IP address to listen on for HTTPS connections
ssl-host = localhost

# The port to listen on for HTTPS connections
ssl-port = 8081

# Private key path
ssl-key = /etc/puppetdb/ssl/private.pem

# Public certificate path
ssl-cert = /etc/puppetdb/ssl/public.pem

# Certificate authority path
ssl-ca-cert = /etc/puppetdb/ssl/ca.pem

certificate-whitelist = /etc/puppetdb/ssl-whitelist

And the ssl-whitelist file:

localhost
handsoff
handsoff.dc0.testing.de

lsof shows me the listening ports

# lsof -i -P | grep puppetdb | grep 80
java     2549      puppetdb   20u  IPv6   4982      0t0  TCP localhost:8080 (LISTEN)
java     2549      puppetdb   34u  IPv6   9353      0t0  TCP localhost:8081 (LISTEN)

In my /etc/hosts I have these lines

127.0.0.1   localhost
127.0.1.1   handsoff handsoff.dc0.testing.de

So from my point of view puppet should be able to connect to puppetdb on port 8081 through localhost and through handsoff. (I tested both, both with the same results)

Any ideas what might be wrong with this setup?

Edit: I also tried to set this in my puppetdb.conf:

[main]
server = localhost
port = 8081

But then I get err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for handsoff.dc0.testing.de to PuppetDB at localhost:8081: hostname does not match the server certificate

I'm using self-signed certificates (didn't change anything on that after I did the install with apt-get). My guess was that the whitelisting would allow me to use the certificate this way. Is that wrong?

Solution

Make sure hostname -f gives full qualified name (if not, update /etc/hostname and update sysctl kernel.hostname={your hostname}).

Check your current certificates:

echo | openssl s_client -connect <puppetdb fqdn>:8081 | openssl x509 -noout -dates

For latest version of PuppetDB (4.2):

1. stop puppetDB service puppetdb stop
2. remove old certificates: rm -rf /etc/puppetlabs/puppetdb/ssl/
3. generate new certificates, puppet DB comes with a handy tool: puppetdb ssl-setup

4. in case you're using SSL connection to DB you might need to update keystore:

   keytool -import -alias "My CA" -file /etc/puppetlabs/puppetdb/ssl/ca.pem -keystore /etc/puppetlabs/puppetdb/ssl/truststore.jks 
   

5. start puppetDB service puppetdb start

Comments

[AWippler]

Can you telnet into the port? telnet localhost 8081

[dawud]

Can the puppet client verify the validity of the CA certificate? is it self-signed? are you using the puppet PKI for puppetdb? what versions of puppet and puppetdb are you using?

[AWippler]

puppetdb is listening on 127.0.0.1 and your puppet master is trying to connect to 127.0.1.1 ... fix this and try again.

[Kai]

I added a few informations about that to me original post.

[byoung]

Try setting host = handsoff or host = 127.0.1.1 in jetty.conf to make PuppetDB listen on that address instead of using localhost in puppetdb.conf.

[leeand00]

I did everything but step 4 and it appears to have worked...