UFW blocking port 80 when it should not

ufw

I have these rules in UFW (default deny):

80                         ALLOW       10.0.0.0/8
443                        ALLOW       10.0.0.0/8
80                         DENY        Anywhere
443                        DENY        Anywhere

I am connecting from a machine on 10.0.0.0/8 to Apache listening on port 80. We noticed nginx running on another machine (a load balancer in front of this apache machine) was occasionally throwing "connect timeout" errors. I am able to recreate the timeout by simply running a for loop in a bash script that hits port 80 from that nginx machine. I get perhaps 3 or 4 timeouts in a batch of 1000 tests.

In /var/log/messages I see these when the timeouts happen:

Dec 1 01:01:01 webserver.mydomain.com kernel: [UFW BLOCK] IN=eth0
OUT= MAC=00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd SRC=10.0.0.5
DST=10.0.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=16923 DF PROTO=TCP
SPT=60064 DPT=80 WINDOW=5792 RES=0x00 SYN URGP=0

Why would UFW be blocking these? If I disable UFW these connection timeouts disappear completely and everything works as expected. Other ports (like 22) don't have this problem, I can run scripts that hammer those endlessly without any issues even with ufw activated.

Best Answer

I spent a significant amount of time trying to troubleshoot this problem. The involvement of UFW was a symptom of the real problem and not the cause. I found a solution so I didn't want to leave the question unanswered.

I discovered that for a reason I cannot yet explain syncookies were disabled on the apache servers behind the load balancer:

# sysctl -a | grep syncookies
net.ipv4.tcp_syncookies = 0

The reason I cannot explain this is that it is set to 1 in the default Centos6 /etc/sysctl.conf file. That is a separate issue for me to figure out.

You can read more about syn cookies here:

http://en.wikipedia.org/wiki/SYN_cookies

These are relatively busy servers that handle a lot of connections. My best guess is that enabling UFW (and thus enabling iptables) slowed things down just enough for the syn queue to fill up and without syncookies on connections started getting refused.

Related Topic