Puppet client caching but not updating local facts

Question

I have a server pmaster-dev that is a puppet client (its master is pmaster). The server pmaster-dev itself acts as a puppet master for several clients.

When pmaster-dev checks in with pmaster it is caching its facts to a local sqlite3 database file /var/lib/puppet/state/clientconfigs.sqlite. On every subsequent check-in pmaster-dev never updates this local cache. Thus, its facts are always stale. Other clients of pmaster (including pmaster itself) never cache.

How do we either tell it to update the cache or disable caching of the facts? Why is it caching while other clients of pmaster are not?

We are running 2.7.18 on a Debian squeeze system.

Here is the /etc/puppet/puppet.conf file for pmaster-dev:

[agent]
server = pmaster.example.org
environment = master
configtimeout = 300

logdir = /var/log/puppet
vardir = /var/lib/puppet
ssldir = /etc/puppet/ssl
rundir = /var/run/puppet

ca_server = puppetca.example.org
ca_port = 8141

graph = true
report = true
pluginsync = true

classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
diff_args = '-u'
show_diff = true


[master]
certname = pmaster-dev.example.org
syslogfacility = local2

logdir = /var/log/puppet
vardir = /var/lib/puppet
rundir = /var/run/puppet
ssldir = /srv/puppetmaster/ssl

reports = tagmail,lastcheck,logcache
manifest = /srv/puppet/$environment/manifests/site.pp
graph = true
modulepath = /srv/puppet/$environment/modules:/srv/puppet/$environment/services:/srv/puppet/$environment/clients
cacrl = /srv/puppetmaster/ssl/crl.pem
ca = false

manifestdir = /srv/puppet/$environment/manifests
queue_source = stomp://pupqueue-dev.example.org:61613/
async_storeconfigs = true
storeconfigs = false
dbadapter = mysql
dbuser = puppet
dbpassword = secret
dbserver = pupqueue-dev.example.org
dbname = puppet

ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Solution

After some digging in the Puppet Ruby source files, I tracked the issue to a bug where there is conflation of Puppet configuration file sections. It all boils down to the word "master"

Summary: Puppet masters when acting as a Puppet client and in the "master" environment cause problems with the configuration file parsing.

Details: When the puppet agent starts one of the things it does is parse the configuration file (normally /etc/puppet/puppet.conf).

The configuration file is broken up into sections, e.g., "[main]", "[agent]", "[master]", etc. Each of these sections is stored separately in the hash generated from parsing the configuration file. In my case, pmaster-dev has these sections: "memory", "master", "cli", and "agent".

There can also be per-environment sections. For example, if there is an environment called "stable201301" then there can be a section in the configuration file headed "[stable201301]".

For each of the above sections (called "sources" in the code) you examine all the settings that have associated "hooks". If in that section the setting with a hook is defined, you call that setting's hook. Some of the settings that have hooks include catalog_format, node_name_fact, and storeconfigs.

One of the more interesting settings is "async_storeconfigs" which has a hook that sets up a cache class.

Recall that the sections can also include ones for each environment. THE PROBLEM: What if the Puppet master (when acting as a Puppet client) is in the "master" environment?

The usual role for the "[master]" section is for settings for a Puppet master. But if the Puppet master itself is using the "master" environment we have a conflict. In particular, the "async_storeconfigs" setting from the "[master]" gets loaded from the configuration file. Since the Puppet master (as a Puppet client) is in the "master" environment this setting causes the cache class to be set and so when puppet agent --test is run the puppet client looks for cached facts.

Solution(?): You could make sure that a Puppet master never used the "master" environment. A better solution would be to have the client configuration parser not look at the "[master]" section (since that is only applicable to Puppet masters). An even betterer solution would be to separate client and master configuration files altogether.

Comments

[Benjamin Bisinger]

When I run /usr/bin/puppet agent --test on pmaster-dev the output starts with info: Connecting to sqlite3 database: /var/lib/puppet/state/clientconfigs.sqlite3, so it appears that the file clientconfigs.sqlite3 is used by the puppet client (at least in this case). If a client is not supposed to be using that file, why is the puppet client on pmaster-dev accessing it?

[rra]

The problem is definitely not due to an initial incomplete configuration and is definitely being used by the client, not the server. If the file is deleted, running puppet agent recreates it (and then stops updating it from that point forward). I have no idea what's going on, though.

[Damien Tournoud]

If you are actually using puppetqd, try moving the db* configuration to [main] instead of [master]. In general, check the configuration that puppetqd is using. You can also try to switch to standard storeconfigs instead of the async_storeconfigs, to see if it fixes your issue.

[Damien Tournoud]

A run of puppet agent will trigger the stored configuration to be updated by the master. That's to be expected. But the update is always done by the master (or by the queue process triggered by the master), this stored configuration database doesn't exist on the clients at all.

[Benjamin Bisinger]

The server pmaster-dev has two puppet roles: it is a puppet client of some other puppet master, and it is a puppet master for some set of clients. What is confusing is that when you run puppet agent --test on pmaster-dev (which should trigger only its client role), you see an attempted update of clientconfigs.sqlite3 on pmaster-dev which should only happen in pamster-dev's master role.