I'm not sure if this is very helpful, but I've struggled a bit with the same problem - and this is what I've concluded;
Hash-based load balancing will, as you've already established, never give you perfect load balancing. The behavior you see can simply be explained by having a few of the most visited / largest pages on the same server - by having few pages that gets a lot of traffic, and a lot of pages that get little traffic, this will be enough to skew the statistics.
Your configuration is to use consistent hashing. The ID's and server weight determine the final server the hashed entry will be directed to - that is why your balancing is affected by this. The documentation is pretty clear that even though this is a good algorithm for balancing caches - it may require you to change around the IDs and increase the total weight of the servers to get a more even distribution.
If you take a large sample of unique addresses (more than 1000), and you visit each of these one time - you should see that the session counter is a lot more equal across the three backends than if you allow 'ordinary' traffic against the balancer as this is affected by the traffic pattern of the site as well.
My advice would be to make sure that you hash the entire URL, not just what's to the left of "?". This is controlled by using balance uri whole
in the configuration. Ref. the haproxy documentation. If you have a lot of URL's which have the same base, but with varying GET-parameters - this will definitely give you improved results.
I would also take into consideration how the load balancing affects the capacity of your cache servers. If it doesn't effectively affect redundancy in any way - I wouldn't worry too much about it, as getting perfect load balancing isn't something you are likely to achieve with URI-hashing.
I hope this helps.
if your haproxy is live on the internet (ie, has public IP, not behind firewall, etc).... you could rate limit further down the stack by using iptables...
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP
so the haproxy daemon wont even get the traffic, which means more efficiency and less load. this would only work if the haproxy has public ip (not behind amazon elb, for example)
Best Answer
This solution requires at least haproxy 1.6.
First, add the following to the frontend:
http-request set-header X-DOS-Protect %[src];%[req.fhdr(host)]%[capture.req.uri]
Then, add the following to the backend:
I was not able to find a way to do the tracking in the frontend as I didn't find a way to apply a converter on that concatenated string that makes up the X-DOS-Protect header.
I am applying the hash function in order to make sure that you don't store a huge string in the stick-table as it could easily lead to a denial of service. If you think that this hash function is not suitable for you due to too many possible collisions, you could also make it bigger by applying crc32 to each of the concatenated components (and, of course, removing it when storing the data and switching to a bigger stick-table storage), like this:
http-request set-header X-DOS-Protect %[src,crc32(1)];%[req.fhdr(host),crc32(1)]%[capture.req.uri,crc32(1)]
Please note that this last solution will use more than 7 times more memory than the first one, for each entry in the stick table. Of course, the collision risk would be way smaller too.