We're interested in restricting the number of requests per second and/or available bandwidth to HTTP clients, to stop accidental DoS. We provide free scientific data and web services, and sadly some users' scripts aren't well behaved.
I know there's lots of Apache mods that allow you to throttle per client IP address, but the problem is, sometimes we see people doing distributed crawling from their clusters (today this caused a load average > 200 incident!).
What I'd really like to do is throttle per /24 subnet, but without having to specify which subnets in advance.
Ideally, I'd also like to be able to do this as a proportion of a maximum cap, so if we're only seeing requests from one subnet, they get to use all the server's resources, but if two subnets are competing, they get to use half each.
Is this possible with either:
- Apache mods
- Traffic control
- Proxy server
- Something else?
Thanks!
EDIT: Couple of further things… If anything needs to be done at the network infrastructure level (e.g. routers) that's out of our responsibility and becomes an instant PITA. So I'm hoping to find a solution that only requires changes at the server level. Also please don't be offended if I take a while to pick a winner, this is a new topic to me so I want to read around the suggestions a bit 🙂
Best Answer
If you are using HAProxy or can use it check see if this blog post helps
</end_shameless_promotion_of_a_fellow_admin_and_company :)>