First, a quick overview of our setup:
- We have a Foreman instance running properly, let's call it foreman.
- The Puppet master works well, let's call it puppetmaster.
- There are two hosts contactable from foreman and puppetmaster with no firewalls in between, let's call them host-a and host-b.
- All hosts are running 3.x
We've started expanding our use of Foreman and want to enable the Run Puppet button. I followed the wiki and got it to work in the initial tests. However, on further testing the button has produced some very inconsistent results. I seem to recall reading that there is a 2 minute splay by default on triggered runs, but that's not the problem here. Sometimes, when I hit the button, a host hops to and starts the run right away. Most of the time however, it never triggers a run.
Foreman has the following in the sudoers config:
Defaults:foreman-proxy !requiretty
foreman-proxy ALL = NOPASSWD: /usr/bin/puppet kick *
The hosts have the following in auth.conf, a blank namespaceauth.conf and listen = true in puppet.conf.
# allow foreman to start puppet runs
path /run
auth any
method save
allow foreman.domain.lan
When attempting to trigger a Puppet run via Foreman, this comes out in the production log.
Started GET "/hosts/host-a.domain.lan/puppetrun" for 192.168.8.48 at 2014-07-15 21:44:59 +0000
Processing by HostsController#puppetrun as HTML
Parameters: {"id"=>"host-a.domain.lan"}
Redirected to http://foreman.domain.lan/hosts/host-a.domain.lan
Completed 302 Found in 48ms (ActiveRecord: 1.3ms)
Started GET "/hosts/host-a.domain.lan" for 192.168.8.48 at 2014-07-15 21:44:59 +0000
Processing by HostsController#show as HTML
Parameters: {"id"=>"host-a.domain.lan"}
Rendered hosts/_overview.html.erb (16.2ms)
Rendered hosts/_metrics.html.erb (0.3ms)
Rendered hosts/show.html.erb within layouts/application (734.1ms)
Rendered home/_user_dropdown.html.erb (2.0ms)
Read fragment views/tabs_and_title_records-2 (0.2ms)
Rendered home/_topbar.html.erb (3.8ms)
Rendered layouts/base.html.erb (5.9ms)
Completed 200 OK in 781ms (Views: 652.3ms | ActiveRecord: 108.6ms)
And that's about it, Puppet runs are occasionally triggered on the hosts but not always. Any ideas?
EDIT: I should add that when run triggers aren't successful, no messages come out on the host's side when running puppet agent –no-daemonize –debug
MOAR EDIT: Running Foreman v1.5.1 plus Puppet 3.6.2 on the puppetmaster and foreman, v3.3.1 on the hosts.
Best Answer
I tried a number of ways to get kick running reliably but couldn't and it's deprecated anyway so I moved on to puppetssh. That required configuring smart-proxy's SSH keys, the foreman-proxy user and its sudo rights. There isn't a lot of documentation on this puppet run method and the DEBUG logging in foreman-proxy doesn't include any command output so it was hard to tell where/how something was failing.
By default, the foreman-proxy user doesn't have a shell (on RHEL) and wouldn't be able to run /usr/bin/ssh. A simple usermod -s /bin/bash did the trick but that seems like a security risk to me. On top of that, different sudo rights are necessary to run it so I had to change foreman-proxy ALL = NOPASSWD: /usr/bin/puppet kick to foreman-proxy ALL = NOPASSWD: /usr/bin/ssh. After that it was just a matter of configuring the ssh command.
In my case, we deployed a user account and SSH keys for running Puppet commands with sudo on client machines. I copied the private key to some place foreman-proxy could use it (eg: /etc/foreman-proxy/puppetssh/id_rsa) and configured the puppetssh parameters in /etc/foreman-proxy/settings.yml like so: