Puppet client caching but not updating local facts

puppetpuppetmaster

I have a server pmaster-dev that is a puppet client (its master is pmaster). The server pmaster-dev itself acts as a puppet master for several clients.

When pmaster-dev checks in with pmaster it is caching its facts to a local sqlite3 database file /var/lib/puppet/state/clientconfigs.sqlite. On every subsequent check-in pmaster-dev never updates this local cache. Thus, its facts are always stale. Other clients of pmaster (including pmaster itself) never cache.

How do we either tell it to update the cache or disable caching of the facts? Why is it caching while other clients of pmaster are not?

We are running 2.7.18 on a Debian squeeze system.

Here is the /etc/puppet/puppet.conf file for pmaster-dev:

[agent]
server = pmaster.example.org
environment = master
configtimeout = 300

logdir = /var/log/puppet
vardir = /var/lib/puppet
ssldir = /etc/puppet/ssl
rundir = /var/run/puppet

ca_server = puppetca.example.org
ca_port = 8141

graph = true
report = true
pluginsync = true

classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
diff_args = '-u'
show_diff = true


[master]
certname = pmaster-dev.example.org
syslogfacility = local2

logdir = /var/log/puppet
vardir = /var/lib/puppet
rundir = /var/run/puppet
ssldir = /srv/puppetmaster/ssl

reports = tagmail,lastcheck,logcache
manifest = /srv/puppet/$environment/manifests/site.pp
graph = true
modulepath = /srv/puppet/$environment/modules:/srv/puppet/$environment/services:/srv/puppet/$environment/clients
cacrl = /srv/puppetmaster/ssl/crl.pem
ca = false

manifestdir = /srv/puppet/$environment/manifests
queue_source = stomp://pupqueue-dev.example.org:61613/
async_storeconfigs = true
storeconfigs = false
dbadapter = mysql
dbuser = puppet
dbpassword = secret
dbserver = pupqueue-dev.example.org
dbname = puppet

ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Best Answer

After some digging in the Puppet Ruby source files, I tracked the issue to a bug where there is conflation of Puppet configuration file sections. It all boils down to the word "master"

Summary: Puppet masters when acting as a Puppet client and in the "master" environment cause problems with the configuration file parsing.

Details: When the puppet agent starts one of the things it does is parse the configuration file (normally /etc/puppet/puppet.conf).

The configuration file is broken up into sections, e.g., "[main]", "[agent]", "[master]", etc. Each of these sections is stored separately in the hash generated from parsing the configuration file. In my case, pmaster-dev has these sections: "memory", "master", "cli", and "agent".

There can also be per-environment sections. For example, if there is an environment called "stable201301" then there can be a section in the configuration file headed "[stable201301]".

For each of the above sections (called "sources" in the code) you examine all the settings that have associated "hooks". If in that section the setting with a hook is defined, you call that setting's hook. Some of the settings that have hooks include catalog_format, node_name_fact, and storeconfigs.

One of the more interesting settings is "async_storeconfigs" which has a hook that sets up a cache class.

Recall that the sections can also include ones for each environment. THE PROBLEM: What if the Puppet master (when acting as a Puppet client) is in the "master" environment?

The usual role for the "[master]" section is for settings for a Puppet master. But if the Puppet master itself is using the "master" environment we have a conflict. In particular, the "async_storeconfigs" setting from the "[master]" gets loaded from the configuration file. Since the Puppet master (as a Puppet client) is in the "master" environment this setting causes the cache class to be set and so when puppet agent --test is run the puppet client looks for cached facts.

Solution(?): You could make sure that a Puppet master never used the "master" environment. A better solution would be to have the client configuration parser not look at the "[master]" section (since that is only applicable to Puppet masters). An even betterer solution would be to separate client and master configuration files altogether.

Related Topic