HAProxy health checks and changing server state

haproxy

I've poured through lots of docs and Google searches but can't seem to find an answer. I'm using HAProxy with several servers that take a while to start–initial loads on my local VM are about a minute or two. It seems from watching the debug logs that HAProxy checks these servers, fails a level 4 connection, then marks the server as permanently dead. Googling doesn't indicate a way to retrigger the health checks.

My assumption was that they'd happen again after a delay, but I don't see any notices in my logs that the servers are being rechecked. If I restart HAProxy it does detect that the servers are up, but ideally it would mark them as dead for a few minutes, detect that they're back up, then bring them back into the rotation. I've also tried sending SIGHUP but this only prints a report of the results of the last check. The config manual is a bit overwhelming. What am I missing?

Here is my haproxy.cfg:

global
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 1024
  user proxy
  group proxy
  tune.ssl.default-dh-param 2048
  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:!NULL:!aNULL:!RC4:!RC2:!MEDIUM:!LOW:!EXPORT:!DES:!MD5:!PSK:!3DES
  debug

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option forwardfor
  option http-server-close
  option redispatch
  retries 3

  timeout connect 5s
  timeout client 50s
  timeout server 50s

frontend http-in
  bind *:80
  bind *:443 ssl crt /etc/ssl/private/perceptronapp.com.pem no-sslv3
  redirect prefix https://www.perceptronapp.com code 301 if { hdr(host) -i perceptronapp.com }
  redirect scheme https if !{ ssl_fc }
  use_backend blog if { path_beg /blog }
  use_backend app

backend app
  balance leastconn
  cookie JSESSIONID insert nocache
  server s1 s1.perceptron.skydns.local:3000 check inter 3m cookie s1
  server s2 s2.perceptron.skydns.local:3000 check inter 3m cookie s2

backend blog
  server s1 s1.ghost.skydns.local:2368 check inter 3m

Best Answer

Figured it out. HAProxy does its DNS resolution at startup, which I'd suspected but had a hard time confirming since all my queries just led me back to the config manual. Since I own the DNS infrastructure and namespace being queried (SkyDNS from Docker containers) I'd hoped to use that for service discovery and HAProxy backends. Seems like an Nginx-like resolver system may be on the HAProxy roadmap according to this, but a checkout and quick scan of the latest sources didn't indicate that was live yet.

Have since switched to confd-based config file generation and graceful restarts. A bit more annoying to configure but seems to do the job.

Related Topic