I'm having trouble with one of our HAProxy-Servers that uses a backend with TLS. When starting HAProxy the backend will report all servers as down:
Server web_remote/apache_rem_1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 41ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
The configuration for the backend is as follows:
backend web_remote
balance leastconn
option httpchk HEAD /
option redispatch
retries 3
default-server inter 5000 rise 2 fall 5 maxconn 10000 maxqueue 50000
server apache_rem_1 192.0.2.1:12345 check maxconn 1000 maxqueue 5000 ssl ca-file /etc/ssl/web.pem
server apache_rem_2 2001:DB8:3:4:5:6:8:12345 check maxconn 1000 maxqueue 5000 ssl ca-file /etc/ssl/web.pem
This backend worked just fine until now, a quick wget
on the server also worked and openssl s_client
reports the certificate of the backend to be valid.
Is there anything else for HAProxy/TLS that could be configured wrong? How could I debug this issue when everything else reports the handshake was successful?
Best Answer
After trying to find ways to debug with HAProxy I finally gave in and used tcpdump to debug the network traffic.
By importing the TLS key and the PCAP file into my SSL-enabled Wireshark, I found the error very quickly: The Root CA was unknown to HAProxy, so the verify was failing. It turned out, a coworker changed the cert but didn't know this PEM needs the complete trust chain. Adding the correct root CA to the .PEM file fixed this and the server is back to normal operations.