Haproxy introduced latency

haproxy

I run an online multiplayer game with about 50K simultaneous players at peak time. The game has a custom client-server protocol (deflate-compressed json blobs with some header info) that can be both client initiated request/response pair or server push. Clients connect to an haproxy instance on AWS, which then balances the load between two application servers (written in Java).

Of late, I notice that at peak hours the response to my clients are quite slow. If I ask a client to connect directly to my Java server, it is as fast as I expect it to be. But if the connection goes through the Haproxy instance, a very large latency is introduced. I measure this latency through a custom Nagios plugin I wrote, and at this moment the values stand:

Direct to server A: 135829 microseconds
Direct to server B: 128737 microseconds
Via Haproxy (balanced to one of these servers): 12990736 microseconds

I am stuck at the moment as to what to do next. I'm running haproxy 1.5.5. Config is here: http://pastebin.com/rQ8y6AQj. I have transparent proxying enabled. net.netfilter.nf_conntrack_max is set to 131072. Everything is running on Amazon EC2.

Best Answer

First of all are you sure the connections are not just hitting the queue? i.e. you have reach maxconns? What does your stats page show?

Also just disable conntrack (it sucks):

iptables -t raw -I PREROUTING -p tcp --dport 8443 -j NOTRACK
iptables -t raw -I PREROUTING -p tcp  --sport 8443 -j NOTRACK
iptables -t raw -I OUTPUT -p tcp --dport 8443 -j NOTRACK
iptables -t raw -I OUTPUT -p tcp --sport 8443 -j NOTRACK
Related Topic