I have a corner case where some application servers behind HaProxy return 500 codes for some requests if they are running an old version of code. Generally this only happens when an upgrade is in progress across the cluster, but can happen if a deploy partially fails too. It is highly likely that another app server in the same pool will be able to service the request.
I wonder if HAProxy can detect that a backend server replied with a 500 error for a specific request (without using a health check), and try again on another server. Sort of like an Option redispatch for errors instead of http timeouts.
It seems like http requests would only go to the client browser, never back to Haproxy, but I thought I'd ask and see if anyone has solved this, maybe even with a WAF or two way proxy.
Best Answer
Yes on HAProxy version >2.0: add
retry-on all-retryable-errorsto the relevant backend.But be careful with it (POST requests may be retried causing duplicate database operations); see https://www.haproxy.com/blog/haproxy-layer-7-retries-and-chaos-engineering/