Apache Load Balancer – Health Check with Long Timeout Configuration

apache-2.4healthcheckload balancing

I'm using Apache HTTP Server as a reverse proxy for a couple of Tomcat instances. I've setup load balancing as follows:

<Proxy "balancer://tomcat-app">
    BalancerMember "http://10.0.0.1:8080" hcmethod=HEAD hcuri=/status
    BalancerMember "http://10.0.0.2:8080" hcmethod=HEAD hcuri=/status
</Proxy>
ProxyPass        "/app" "balancer://tomcat-app"
ProxyPassReverse "/app" "balancer://tomcat-app"

The problem is that the tomcat containers take around 15 minutes each to restart, due to the app taking as much to be redeployed. Ideally, during this time the load balancer would detect that one of the backend servers is offline and temporarily send all incoming requests to the other, healthy backend server. Unfortunately, I have another line in my httpd.conf:

ProxyTimeout 600

This is apparently needed because the app can legitimately take as much to respond to some requests. The consequence is, the load balancer is unable to detect that the app is not "ready" in less than 10 minutes.

Question: Is there a way to set a different timeout for the healthcheck than for the proxied requests?

Note: Any suggestion on how to better approach this scenario will be welcome.

Best Answer

My problem was not about the timeout: the health check started to work as desired once I set a ProxyHCExpr.

I noticed this by looking at the requests made to the status page on the backend nodes: only after I defined a ProxyHCExpr did they arrive more or less regularly every second. Note that I don't need to use the expression, adding hcexpr=23, but an expression needs to be set anyway in order for mod_proxy_hcheck to do its job. I would have expected that either it could be safely omitted or that apachectl -t would throw an error or warning.

In the docs, it says:

hcexpr Name of expression, created via ProxyHCExpr, used to check response headers for health. If not used, 2xx thru 3xx status codes imply success

Based on this, I had wrongly assumed that setting ProxyHCExpr would also be optional. Unfortunately, it was not working as desired until I set it. My configuration file went from:

<Proxy "balancer://tomcat">
    BalancerMember "http://10.0.0.1:8080" hcmethod=HEAD hcinterval=1 hcpasses=9 hcuri=/app/status
    BalancerMember "http://10.0.0.2:8080" hcmethod=HEAD hcinterval=1 hcpasses=9 hcuri=/app/status
</Proxy>

To:

ProxyHCExpr 23 {%{REQUEST_STATUS} =~ /^[23]/}
<Proxy "balancer://ifis-tomcat-col">
    BalancerMember "http://10.0.0.1:8080" hcmethod=HEAD hcinterval=1 hcpasses=9 hcuri=/app/status
    BalancerMember "http://10.0.0.2:8080" hcmethod=HEAD hcinterval=1 hcpasses=9 hcuri=/app/status
</Proxy>

This was the configuration as seen from the balancer-manager:

MaxMembers  StickySession   DisableFailover Timeout FailoverAttempts    Method      Path    Active
2 [2 Used]  (None)          Off             0       1                   bybusyness  /app    Yes

Worker URL          Route   RouteRedir  Factor  Set Status  Elected Busy    Load    To  From    HC Method   HC Interval Passes  Fails   HC uri      HC Expr
http://10.0.0.1:8080                    1.00    0   Init Ok 0       0       0       0   0       HEAD        1000ms      9 (0)   1 (0)   /app/status
http://10.0.0.2:8080                    1.00    0   Init Ok 0       0       0       0   0       HEAD        1000ms      9 (0)   1 (0)   /app/status

Then it became:

MaxMembers  StickySession   DisableFailover Timeout FailoverAttempts    Method      Path    Active
2 [2 Used]  (None)          Off             0       1                   bybusyness  /app    Yes

Worker URL          Route   RouteRedir  Factor  Set Status  Elected Busy    Load    To  From    HC Method   HC Interval Passes  Fails   HC uri      HC Expr
http://10.0.0.1:8080                    1.00    0   Init Ok 0       0       0       0   0       HEAD        1000ms      9 (0)   1 (0)   /app/status 
http://10.0.0.2:8080                    1.00    0   Init Ok 0       0       0       0   0       HEAD        1000ms      9 (0)   1 (0)   /app/status 

Health check cond. expressions:
Expr name   Expression
23          %{REQUEST_STATUS} =~ /^[23]/
Related Topic