Linux – Why is UFW blocking some allowed internal connections

Notice that your packet has both the FIN and ACK bits set. This is the last packet that the remote host sends in the TCP tear down (end of connection) procedure.

What happens is, when your host has finished sending it sets the FIN and ACK flags on the last packet. The remote hosts sends a packet with ACK set followed by a packet with FIN and ACK set.

Local          remote
FIN ACK ---->
        <----  ACK
        <----  FIN ACK (?optional?)
ACK     ----->

In practice, the remotes FIN ACK is considered optional so the netfilter firewall will flush it's connection table when it sees the ACK so when the FIN ACK packet arrives it has no associated connection and is dropped.

I spent a significant amount of time trying to troubleshoot this problem. The involvement of UFW was a symptom of the real problem and not the cause. I found a solution so I didn't want to leave the question unanswered.

I discovered that for a reason I cannot yet explain syncookies were disabled on the apache servers behind the load balancer:

# sysctl -a | grep syncookies
net.ipv4.tcp_syncookies = 0

The reason I cannot explain this is that it is set to 1 in the default Centos6 /etc/sysctl.conf file. That is a separate issue for me to figure out.

You can read more about syn cookies here:

http://en.wikipedia.org/wiki/SYN_cookies

These are relatively busy servers that handle a lot of connections. My best guess is that enabling UFW (and thus enabling iptables) slowed things down just enough for the syn queue to fill up and without syncookies on connections started getting refused.