Foreman displays a server as “out of sync”, what could be the reason and what is the right way to troubleshoot such an issue

foremanpuppetpuppetmaster

There's a puppet server in the company I work for with Foreman installed.

There are around 200 Linux machines which are being managed by the Puppet server.

When looking in the Foreman GUI, most server appear with a green icon saying "No changes" but there are some servers which appear with a yellow/orange icon with a tooltip saying "out of sync".

You can see in the following screen shot that most servers appear with green status and only server "sgproxy08" is out of sync:
enter image description here

Running puppet agent -t on the sgproxy08 server finishes properly:

[root@sgproxy08 ~]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for sgproxy08.sg.company.com
Info: Applying configuration version '1436862601'
Notice: Finished catalog run in 49.90 seconds
[root@sgproxy08 ~]#

Now there are two questions:

What are the requirements for this icon to become green and in sync?
How can I troubleshoot such an issue?

Best Answer

If you're on Foreman 1.8 or lower, your host will be shown as 'out of sync' only when Foreman has not received any reports for that host for 5 minutes longer than the setting 'puppet_interval' (check it in Administer > Settings). By default this would be 35 minutes. There's an 'outofsync_interval' setting that allows you to customize this on Foreman 1.9.

In order to start troubleshooting this (missing reports), I would look at /var/log/foreman/production.log on your Foreman boxes and check if you receive the report from that host when Puppet finishes its run.

Related Solutions

Puppet – Not Signing Cert or Getting New Requests

Any open signing requests should be listed in puppet cert list on the master. If they don't show up there, there's no use trying to sign them. Once signed, they disappear from the list and will only show up in puppet cert list --all.

It appears your master is not receiving signing requests from the agent, since your list is empty There is a number of things that could be wrong. Let's start with these:

Is the master running?
Does the hostname 'puppet' or 'puppet.abc.com' resolve from the agent?
Is TCP port 8140 on the master reachable from the agent (try: telnet puppet 8140)?
What does syslog on the agent say?
Try puppet agent --test on the agent, which will attempt to connect to the master and stay in foreground to show the output.

“Run Puppet” function in Foreman works inconsistently

I tried a number of ways to get kick running reliably but couldn't and it's deprecated anyway so I moved on to puppetssh. That required configuring smart-proxy's SSH keys, the foreman-proxy user and its sudo rights. There isn't a lot of documentation on this puppet run method and the DEBUG logging in foreman-proxy doesn't include any command output so it was hard to tell where/how something was failing.

By default, the foreman-proxy user doesn't have a shell (on RHEL) and wouldn't be able to run /usr/bin/ssh. A simple usermod -s /bin/bash did the trick but that seems like a security risk to me. On top of that, different sudo rights are necessary to run it so I had to change foreman-proxy ALL = NOPASSWD: /usr/bin/puppet kick to foreman-proxy ALL = NOPASSWD: /usr/bin/ssh. After that it was just a matter of configuring the ssh command.

In my case, we deployed a user account and SSH keys for running Puppet commands with sudo on client machines. I copied the private key to some place foreman-proxy could use it (eg: /etc/foreman-proxy/puppetssh/id_rsa) and configured the puppetssh parameters in /etc/foreman-proxy/settings.yml like so:

:puppetssh_command: /usr/bin/sudo /usr/bin/puppet agent -t --no-usecacheonfailure
:puppetssh_user: puppetssh
:puppetssh_keyfile: /etc/foreman-proxy/puppetssh/id_rsa

Best Answer

Related Solutions

Puppet – Not Signing Cert or Getting New Requests

“Run Puppet” function in Foreman works inconsistently

Related Topic