Linux – How to read the number of pending requests in the TCP stack backlog under linux

I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:

Jun  2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.

The next step was getting the kernel to recycle all those connections in the TIME_WAIT state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle or increasing ip_conntrack_max to be larger than the number of local ports made available for connections by ip_local_port_range. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle since the docs imply that that is dangerous.

With this configuration I can run ab all day and never run out of connections:

net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768    61000

The tcp_max_orphans setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT state once there were 8192 of them but it doesn't do that for me.

FIN_WAIT (timeout for the FIN request acknowledgement) is not the same as TIME_WAIT (time to ensure that the socket is really not used anymore). And yes, with 65k ports in TIME_WAIT state, you only will be running out of TCP ports if you are using a single requester IP - as might be the case if all your clients are behind a NAT device. You also might be running of resources due to an overly populated transmission control block table - see this excellent even if somewhat dated paper for possible performance implications.

If you are really concerned about your sockets in TIME_WAIT state and do not have stateful firewalls between your clients and your server, you might consider setting /proc/sys/net/ipv4/tcp_tw_recycle, but you would sacrifice RFC compliance and might have interesting side-effects in the future due to that matter.