I'm using Apache HTTP Server as a reverse proxy for a couple of Tomcat instances. I've setup load balancing as follows:
<Proxy "balancer://tomcat-app">
BalancerMember "http://10.0.0.1:8080" hcmethod=HEAD hcuri=/status
BalancerMember "http://10.0.0.2:8080" hcmethod=HEAD hcuri=/status
</Proxy>
ProxyPass "/app" "balancer://tomcat-app"
ProxyPassReverse "/app" "balancer://tomcat-app"
The problem is that the tomcat containers take around 15 minutes each to restart, due to the app taking as much to be redeployed. Ideally, during this time the load balancer would detect that one of the backend servers is offline and temporarily send all incoming requests to the other, healthy backend server. Unfortunately, I have another line in my httpd.conf
:
ProxyTimeout 600
This is apparently needed because the app can legitimately take as much to respond to some requests. The consequence is, the load balancer is unable to detect that the app is not "ready" in less than 10 minutes.
Question: Is there a way to set a different timeout for the healthcheck than for the proxied requests?
Note: Any suggestion on how to better approach this scenario will be welcome.
Best Answer
My problem was not about the timeout: the health check started to work as desired once I set a
ProxyHCExpr
.I noticed this by looking at the requests made to the status page on the backend nodes: only after I defined a
ProxyHCExpr
did they arrive more or less regularly every second. Note that I don't need to use the expression, addinghcexpr=23
, but an expression needs to be set anyway in order for mod_proxy_hcheck to do its job. I would have expected that either it could be safely omitted or thatapachectl -t
would throw an error or warning.In the docs, it says:
Based on this, I had wrongly assumed that setting
ProxyHCExpr
would also be optional. Unfortunately, it was not working as desired until I set it. My configuration file went from:To:
This was the configuration as seen from the balancer-manager:
Then it became: