Removing old node from puppetdb

Any open signing requests should be listed in puppet cert list on the master. If they don't show up there, there's no use trying to sign them. Once signed, they disappear from the list and will only show up in puppet cert list --all.

It appears your master is not receiving signing requests from the agent, since your list is empty There is a number of things that could be wrong. Let's start with these:

• Is the master running?
• Does the hostname 'puppet' or 'puppet.abc.com' resolve from the agent?
• Is TCP port 8140 on the master reachable from the agent (try: telnet puppet 8140)?
• What does syslog on the agent say?
• Try puppet agent --test on the agent, which will attempt to connect to the master and stay in foreground to show the output.

I got it going, but can't say exactly what steps were necessary or not.

This issue started because authentication on several hosts was slow or hanging, and appeared to be related to domain controller/DNS cache issues. Removing domain mydomain.com entry from /etc/resolv.conf on the puppet master and agents solved the issue, but that created issues with existing puppet certs. I ran puppet cert clean --all on the master to try and recreate all certs, but this did not play well with PuppetDB.

Solution

Clean out old certs on master:

puppet cert clean --all

Clean out old certs on all agents:

rm -rf /var/lib/puppet/ssl

Recreate PuppetDB keystores:

facter fqdn is not available after removing domain foo.com from /etc/resolv.conf. This causes puppetdb-ssl-setup to fail silently.

Edit /usr/sbin/puppetdb-ssl-setup, add a piece of code to use just facter hostname if facter fqdn is empty:

# near line 10
fqdn=`facter fqdn`
# add this "if" section
if [ ! -n "$fqdn" ] ; then
  fqdn=`facter hostname`
fi

Permissions fix:

chown -R puppetdb:puppetdb /etc/puppetdb/ssl

Update passwords in /etc/puppetdb/conf.d/jetty.ini with new keystore/truststore passcode (same pass), which you can get from:

cat /etc/puppetdb/ssl/puppetdb_keystore_pw.txt

Restart puppetdb

service puppetdb restart

Then go to each agent and request new certs and sign each on the master.