“Run Puppet” function in Foreman works inconsistently

foremanpuppet

First, a quick overview of our setup:

  1. We have a Foreman instance running properly, let's call it foreman.
  2. The Puppet master works well, let's call it puppetmaster.
  3. There are two hosts contactable from foreman and puppetmaster with no firewalls in between, let's call them host-a and host-b.
  4. All hosts are running 3.x

We've started expanding our use of Foreman and want to enable the Run Puppet button. I followed the wiki and got it to work in the initial tests. However, on further testing the button has produced some very inconsistent results. I seem to recall reading that there is a 2 minute splay by default on triggered runs, but that's not the problem here. Sometimes, when I hit the button, a host hops to and starts the run right away. Most of the time however, it never triggers a run.

Foreman has the following in the sudoers config:

Defaults:foreman-proxy !requiretty
foreman-proxy   ALL = NOPASSWD: /usr/bin/puppet kick *

The hosts have the following in auth.conf, a blank namespaceauth.conf and listen = true in puppet.conf.

# allow foreman to start puppet runs
path /run
auth any
method save
allow foreman.domain.lan

When attempting to trigger a Puppet run via Foreman, this comes out in the production log.

Started GET "/hosts/host-a.domain.lan/puppetrun" for 192.168.8.48 at 2014-07-15 21:44:59 +0000
Processing by HostsController#puppetrun as HTML
  Parameters: {"id"=>"host-a.domain.lan"}
Redirected to http://foreman.domain.lan/hosts/host-a.domain.lan
Completed 302 Found in 48ms (ActiveRecord: 1.3ms)


Started GET "/hosts/host-a.domain.lan" for 192.168.8.48 at 2014-07-15 21:44:59 +0000
Processing by HostsController#show as HTML
  Parameters: {"id"=>"host-a.domain.lan"}
  Rendered hosts/_overview.html.erb (16.2ms)
  Rendered hosts/_metrics.html.erb (0.3ms)
  Rendered hosts/show.html.erb within layouts/application (734.1ms)
  Rendered home/_user_dropdown.html.erb (2.0ms)
Read fragment views/tabs_and_title_records-2 (0.2ms)
  Rendered home/_topbar.html.erb (3.8ms)
  Rendered layouts/base.html.erb (5.9ms)
Completed 200 OK in 781ms (Views: 652.3ms | ActiveRecord: 108.6ms)

And that's about it, Puppet runs are occasionally triggered on the hosts but not always. Any ideas?

EDIT: I should add that when run triggers aren't successful, no messages come out on the host's side when running puppet agent –no-daemonize –debug

MOAR EDIT: Running Foreman v1.5.1 plus Puppet 3.6.2 on the puppetmaster and foreman, v3.3.1 on the hosts.

Best Answer

I tried a number of ways to get kick running reliably but couldn't and it's deprecated anyway so I moved on to puppetssh. That required configuring smart-proxy's SSH keys, the foreman-proxy user and its sudo rights. There isn't a lot of documentation on this puppet run method and the DEBUG logging in foreman-proxy doesn't include any command output so it was hard to tell where/how something was failing.

By default, the foreman-proxy user doesn't have a shell (on RHEL) and wouldn't be able to run /usr/bin/ssh. A simple usermod -s /bin/bash did the trick but that seems like a security risk to me. On top of that, different sudo rights are necessary to run it so I had to change foreman-proxy ALL = NOPASSWD: /usr/bin/puppet kick to foreman-proxy ALL = NOPASSWD: /usr/bin/ssh. After that it was just a matter of configuring the ssh command.

In my case, we deployed a user account and SSH keys for running Puppet commands with sudo on client machines. I copied the private key to some place foreman-proxy could use it (eg: /etc/foreman-proxy/puppetssh/id_rsa) and configured the puppetssh parameters in /etc/foreman-proxy/settings.yml like so:

:puppetssh_command: /usr/bin/sudo /usr/bin/puppet agent -t --no-usecacheonfailure
:puppetssh_user: puppetssh
:puppetssh_keyfile: /etc/foreman-proxy/puppetssh/id_rsa
Related Topic