HAProxy Debian – Achieving HAProxy Graceful Reload with Zero Packet Loss

debianhaproxy

I'm running an HAProxy load balancing server to balance load to multiple Apache servers. I need to reload HAProxy at any given time in order to change the load balancing algorithm.

This all works fine, except for the fact that I have to reload the server without losing a single packet (at the moment a reload is giving me 99.76% success on average, with 1000 requests per second for 5 seconds). I have done many hours of research about this, and have found the following command for "gracefully reloading" the HAProxy server:

haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

However, this has little or no effect versus the plain old service haproxy reload, it's still dropping 0.24% on average.

Is there any way of reloading the HAProxy config file without a single dropped packet from any user?

Best Answer

According to https://github.com/aws/opsworks-cookbooks/pull/40 and consequently http://www.mail-archive.com/haproxy@formilux.org/msg06885.html you can:

iptables -I INPUT -p tcp --dport $PORT --syn -j DROP
sleep 1
service haproxy restart
iptables -D INPUT -p tcp --dport $PORT --syn -j DROP

This has the effect of dropping the SYN before a restart, so that clients will resend this SYN until it reaches the new process.

Related Solutions

Linux – HAProxy health check

The only way to do this with haproxy is create a script that haproxy's health check checks. In the script you are checking the load average and if it's a valid load.. return a response code of 200. If it is too high return a 404 or 500 error. Your haproxy check would look something like

option httpchk HEAD /health_check.php HTTP/1.0

if haproxy sees any 2xx/3xx response it will be valid.. anything else it will take that node offline till the node's load goes down and the health_check.php scripts returns a 200

Restarting Haproxy Gracefully

While troubleshooting this, try to use "nbproc 1" instead of "nbproc 4", increasing the number of processes is rarely needed, and could potentially lead to issues in certain cases.

However, this is most likely not the main problem. The scenario you describe should actually work as you expect.

HAProxy's hot reconfiguration actually works like this :

A new process is started with the specified configuration
The old process receives a SIGTTOU signal so that the new process can take over the port(s)
The old process finishes it's current work and then exits
The new process handles traffic over the port(s)

If anything goes wrong after 2, the old process instead receives a SIGTTIN signal, and continues as before.

Some additional detail in your question could make troubleshooting this problem easier:

What parameters do you use with ab
After a hot reconfiguration, can you connect to the new process with any other clients or not
Which (major) version of HAProxy are you using

Best Answer

Related Solutions

Linux – HAProxy health check

Restarting Haproxy Gracefully

Related Topic