HAProxy Error When Server Goes Down Before Health Check – Solution Guide

Let's say I have web01 in my list of backend servers, and web01 goes down. It will take a few seconds for HAProxy to notice the server is down (depending on the health check interval and how long the timeout is) and take it out of rotation. If a request comes in before that's happened, the client will end up receiving a 503 Service Unavailable error.

What I'd like to happen is to have HAProxy automatically re-try that same request again on another server. I realize the request would end up being really slow, but it would end up resulting in a success rather than an error.

Is there a way to configure HAProxy to retry the HTTP request on another server instead of erroring out? Ideally, I don't ever want a client to receive an error if there's any working servers in the cluster.

Here's my haproxy.cfg:

global
        maxconn 4096
        debug

defaults
        mode    http
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

frontend http-in
        bind *:80
        acl service1 path_reg ^/service1/
        acl service2 path_reg ^/service2/
        use_backend service1 if service1
        use_backend service2 if service2

backend service1
        server web01 127.0.0.1:85 check
        server web02 127.0.0.1:86 check
        reqrep ^([^\ :]*)\ /service1/(.*)     \1\ /\2

backend service2
        server web03 127.0.0.1:87 check
        server web04 127.0.0.1:88 check
        reqrep ^([^\ :]*)\ /service2/(.*)     \1\ /\2