Our infrastructure consists of load balancers running HAProxy and Apache, which forward traffic to our app servers running just Apache. The past few days, we've been seeing connection floods which the load balancers happily pass along, but the connections quickly overwhelm our application servers. They become unresponsive, and our only mitigation tactic is to launch more application servers to sustain the flood. At first we couldn't identify why these servers would go down because there were no real spikes traffic on the load balancers, but after some investigation we see that the number of Apache connections is through the roof.
Attached are some graphs which seem to be the only indicators of the flood. I've lowered the HAProxy maxconn directive for each backend server to a more reasonable number (its previous default was 255), but I worry that new legitimate connections will be delayed until the flood subsides. The service will appear to be up for users with existing connections, but it will appear down to new connections and external monitoring services because HAProxy is rate-limiting without regard to host.
Is there anything else we can do on our end (HAProxy or Apache) to help us sustain the load, or should we look at filtering it outside our network? Since it's such a small bit of traffic, I feel like there is something more we can be doing, but I don't have much familiarity with all of the capabilities of HAProxy. I'm also interested in exploring if HAProxy can rate-limit based on IP, but I haven't found anything.
EDIT: We audited the access log for one of the load balancers, and the request counts for the top IPs are as follows:
1070 69.64.*.*
1227 1.9.*.*
1235 64.71.*.*
1376 69.64.*.*
1459 12.189.*.*
1572 1.9.*.*
1678 208.106.*.*
1982 5.15.*.*
2630 23.22.*.*
3300 76.125.*.* (our office)
3543 216.38.*.*
So, we could dynamically ban IPs that establish too many sessions within a small window, but we couldn't block based on total requests as that would catch us as well. Does this route make sense? Should we do this at the iptables level on the load balancers?
Any advice is much appreciated!
Thanks,
Chris
Best Answer
first, if what you get are "just" connection floods, setting the maxconn on the servers will be enough because haproxy will only pass valid requests to the servers, not plain connections with no request. Also, having a lower maxconn makes your servers faster and in general serves your user faster (provided you don't go too low, keep at least 2-3 times the number of CPU cores on the apache machines).
If you get request floods, then you need better counter measures. Haproxy 1.5-dev has a number of these, it is able to blacklist some IP addresses which are doing too many requests, too high connection rates, too many concurrent connections, or experiencing too many errors.
There is a great explanation here http://blog.exceliance.fr/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/ on how to set this up.
Hoping this helps !