Logging HAProxy Connection Limiting

haproxyrate-limiting

I have some simple connection and connection rate limiting in HAProxy:

    # Store IPs to limit connection rate
    stick-table type ip size 200k expire 5m store gpc0,conn_cur,conn_rate(10s)
    tcp-request connection track-sc0 src

    # Abusers are immediately rejected
    tcp-request connection reject if { sc0_get_gpc0 gt 0 }

    # Test connection count and rate
    acl connabuse sc0_conn_cur gt 20
    acl connkill  sc0_inc_gpc0 gt 0
    tcp-request connection reject if connabuse connkill

    acl rateabuse sc0_conn_rate gt 30
    acl ratekill  sc0_inc_gpc0 gt 0
    tcp-request connection reject if rateabuse ratekill

Unfortunately, this has been causing some problems. I'd like to temporarily stop blocking people, but log the time and IP when those limits are hit, so I can play around with the rules and see what works and what doesn't. How can I do this?

Best Answer

Instead of blocking abusers you could add response headers indicating client IP address that broke the rules. Something like this:

http-request add-header X-Haproxy-Abuse %ci if connabuse
http-request add-header X-Haproxy-Kill %ci if connkill

where %ci represents Client IP address, you can log any other param mentioned here: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4

Than you capture that headers and put them in logs as described in docs: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#capture%20response

capture response header X-Haproxy-Abuse 15
capture response header X-Haproxy-Kill len 15

Related Solutions

Nginx or HAProxy – How to Rate-Limit Concurrent Sessions

There's amazingly nothing right now to report the number of entries in a stick-table for use with ACLs, though that's probably a 10-lines patch to add to haproxy 1.5 if that can help.

If you happen to do it yourself, please send it to me and I'll merge it into mainline. Otherwise ping me off-list so that I don't forget to add this.

Way to rate limit connections with HAProxy using multiple thresholds

I was just trying to do this myself, was having no luck, and decided to resort to my google-fu. The top result for me when looking for multiple levels of rate limiting was this, and I got really excited. Then I saw it had no answers and initially fell into an existential pit of despair. After digging myself out, I kept hacking, and by some stroke of luck, I seem to have figured out how to do it at least for what I needed. Maybe it will work for you too.

Haproxy is really, really cool, and I'm excited to start using it in place of our current load balancing solution, but stick-tables are a bit of a monster to wrap your head around. On that front, I've found one general principle that seems to be helping me, and that's to explicitly refer to every stick table by name when you're trying to do a setup with multiple stick tables. The default behaviour, where the name is implicit (assumed to be the backend you're in), is great... except when you start trying to get fancy with multiple stick tables. So that's why in my configuration below, some of it is more verbose than it has to be. I just find it easier to follow the logic that way. Anyway, here goes (note that this is counting based on cookies for a Moodle application, not the IP, and it's using v1.5.11 of haproxy):

backend dynamic_60
  stick-table type string len 36 size 1m store gpc0_rate(60s)

backend dynamic
  stick-table type string len 36 size 1m store gpc0_rate(10s)
  stick on cookie(MoodleSession) table dynamic
  stick on cookie(MoodleSession) table dynamic_60
  tcp-request content track-sc0 cookie(MoodleSession) table dynamic
  tcp-request content track-sc1 cookie(MoodleSession) table dynamic_60

  acl rate_10s sc0_inc_gpc0(dynamic) gt 0
  acl rate_60s sc1_inc_gpc0(dynamic_60) gt 0
  tcp-request content reject if rate_10s rate_60s FALSE

So, what this is doing is setting one counter to record the rate per 10s and another to record the rate per 60s. Note that it's not actually using these counters to do any rate limiting yet. But you can verify via:

echo "show table dynamic" | socat /var/run/haproxy/admin.sock stdio
echo "show table dynamic_60" | socat /var/run/haproxy/admin.sock stdio

That the rate counters are being maintained separately.

I wanted to find out the minimal configuration I needed to get those counters to actually increment, which is why you see "FALSE" at the end of the "tcp-request content reject" statement. Just defining the acls with the counters won't get them to increment. You have to actually use the acl. Putting "FALSE" at the end simply allows me to use the acl without ever satisfying the condition to actually reject the request. I'll probably just take out the "FALSE" once I decide on some really numbers for those acls.

The real key to getting multiple stick tables to work seems to be doing the "stick on", "track-sc{0|1|2}", and acl definitions using "sc{0,1,2}_inc_gpc0" in the backend where you're actually handling the request. Moving any of those to the dynamic_60 backend caused the count for that to stop working. I guess the reasoning is that it makes no sense to track or apply acls to a backend that's not serving requests because it doesn't actually have the requests going through to pull information from. That said, I'm sure others will have better explanations. I'm pretty new to haproxy.

The next question I asked was: am I limited to just tracking 3 things (as the "track-sc" configuration settings only go from 0-2). I believe that, yes, you can only track the three things. But importantly, it's 3 things per backend that actually serves a request. So, for example, if like me you want to do different rate limiting for static content than dynamic content, you can make your decision on whether to go to a "static" or "dynamic" backend in your frontend, based on something in the request. Then in the "static" backend, you define track-sc0 and track-sc1 on the "static" and "static_60" backends (if you happened to be following a similar naming scheme to the config I put above). Then you'll have 4 stick tables to use to make rate limiting decisions. 10s and 60s rates for both dynamic and static content. Use the 3rd counter, and I'd think you could get your 3 levels in, but I think that would be the limit.

Best Answer

Related Solutions

Nginx or HAProxy – How to Rate-Limit Concurrent Sessions

Way to rate limit connections with HAProxy using multiple thresholds

Related Topic