Mysterious HAProxy request errors

haproxy

I see tons of request errors on one of my frontend proxies (on the order of a few per second), but I can't figure out what's causing them. I've tried using the "show errors" command on the stats socket…

echo "show errors" | socat unix-connect:/var/run/haproxy.stat stdio

But this returns nothing. Debug logging doesn't give me any hints either. Is there some other place I should be looking?

Edit: Just to clarify, there is no "error message" per se (though it would sure help to have one). I'm just looking at the counter labeled "request errors" in the web interface and the socat output, and I'm trying to figure out what's incrementing it.

Best Answer

Have you looked at dmesg? A common problem with Proxy servers is to hit the max Linux connection tracking since each request is using to connections. If this is the case you will see ip_conntrack: table full, dropping packet. in dmesg. You can see the current count and raise it via sysctl or proc:

[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_max
131072
[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_count
185

You can also bypass connection tracking with the NOTRACK target, i.e.:

sudo /sbin/iptables -t raw -A PREROUTING -p tcp --sport 80 -j NOTRACK

Keep in mind that it is a security risk to disable tracking though, you don't want to do it unless you are already behind a stateful firewall.

Can you post the errors you are seeing?

Related Topic