Centos – Philosophy of scaling number of network connections with Centos 5.5

centosnetworking

This morning I received this is my dmesg log:

[3184815.656881] nf_conntrack: table full, dropping packet.
[3184821.442351] net_ratelimit: 282 callbacks suppressed

FWIW, here's the current max:

cat /proc/sys/net/netfilter/nf_conntrack_max 
65536

So, I can gather that the server is dropping TCP connections because it can't keep track of them all. I know I can just set a new max number in sysctl.conf or echo it to /proc/sys/net/netfilter/nf_conntrack_max but I'm not sure that's the best course to take.

Would I be setting up the kernel for certain death? Is there a better approach to handling LOTS of network connections?

Thanks in advance

Best Answer

In terms of nf_conntrack_max, you are only limited by the amount of available kernel memory. Kernel memory isn't swappable so it is actual RAM.

TLDR: On modern systems I set this to 128k with seemingly no ill effect.

Another tunable that you may wish to tweak, however, is /proc/sys/net/netfilter/nf_conntrack_buckets. Every time a packet is examined it is hashed into one of the hashsize buckets (based on src/dst IP and port). Each bucket contains a linked list that then has to be searched for the particular connection. Increasing the number of buckets should (up to a certain point) decrease the number a nodes in the linked list that need to be traversed and therefore increase performance. The tradeoff is that the more buckets you have the higher the probability of empty buckets. An empty bucket takes kernel memory. Thats why you don't set hashsize = nf_conntrack_max. I was unable to find any recommendations on setting the hashsize except the statement "Under normal circumstances ip_conntrack_max equals 8 * hashsize."

Another thing to note is that when your connections are short-lived (e.g., a web server) you may have many connections in the TIME_WAIT state with corresponding entries in the conntrack hash. cat /proc/net/nf_conntrack | grep TIME | wc -l (ip_conntrack on older kernels) or iptstate should tell you this information. To reduce the timeout for these change /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait. You'd probably want to change /proc/sys/net/ipv4/tcp_fin_timeout as well.

Finally, if you really need performance and you can't afford the RAM you could disable the firewall completely and put a dedicated firewall in from of the server.

Related Topic