I am currently trying to setup a load balanced puppet infrastructure. I am using puppetserver for the puppet server(s) and puppet agent 3.7.3 for the clients.
Currently I have four servers setup in a local DNS server and name resolution is working correctly (I have not shown the FDQN's for brevity)
vmhgmaasdns01 IN A 192.168.207.208
vmhgmaasmgmt01 IN A 192.168.207.210
vmhgmaasproxy01 IN A 192.168.207.209
vmhgmaaspuppetdb01 IN A 192.168.207.206
mgmt IN CNAME vmhgmaasproxy01
All of the servers are able to connect to the puppet server vmhgmaasmgmt01 when doing a puppet run using –server=vmhgmaasmgmt01
However when trying to use –server=mgmt I am getting errors
puppet agent --no-daemonize --no-splay --verbose --onetime --server=mgmt
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: end of file reached
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/pluginfacts: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/plugins: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Loading facts
Error: Could not retrieve catalog from remote server: Broken pipe
I can run
openssl s_client -connect mgmt:8140 -cert /etc/puppet/ssl/certs/vmhgmaasproxy01.pem -key ssl/private_keys/vmhgmaasproxy01.pem -CAfile ssl/certs/ca.pem
Which shows I can successfully verify the SSL connection through the load balancer
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-SHA256
Session-ID: 54C623F7CC73B6CEDFC8C6BF1366FE96049030E60667FE170113D30EA2221F06
Session-ID-ctx:
Master-Key: B19715E32AE17A2C7D501D80A9D695C476A99CFB5441D07142650689CD554418C193505A5468364A7E0F482304F32C1E
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1422271478
Timeout : 300 (sec)
Verify return code: 0 (ok)
I have setup /etc/puppet/puppet.conf options to point to the certificates and keys used in the above example.
In HAProxy I have the front-end and back-ends set in TCP mode and when I try to run the puppet run I can see the sessions increasing in the front-end and back-end in the stats page. I can also see that the requests are being received by the puppet server using tcpdump.
The puppetserver certificate have been generated with dns_alt_names for the mgmt loadbalancer as well as the dns name for the host.
I can see nothing in the log files in /var/log/puppetserver/puppetserver.log with log level set to DEBUG for the failed connections.
All servers are running CentOS 6.6 and I have turned SELinux into permissive mode.
Any help greatly appreciated as I have tried and failed over the last three days to follow details from the sparse official documentation
Best Answer
I was able to resolve the issue by setting up the proxy server to use SSL termination to enable layer 7 load balancing and http mode.
This allows the catalogs to goto either of the two puppet servers but certificate requests are routed to the CA. I had to modify webserver.conf to use port and host instead of ssl-host and ssl-port and add the following to master.conf
biggest PITA was not realizing that I had to combine the key and certs for HAproxy into a PEM file. (not in all of the above I have added . to obfuscate the real ips)