I have a node in our cluster which gets lots of "nf_conntrack: table full, dropping packet" messages in the syslog. I checked the nf_conntrack_count and it was running right up against the nf_conntrack_max. Looking into the table, I saw most of the entries were DNS requests, so I added these rules to the "raw" netfilter table.
$ sudo iptables -t raw -vnL
Chain PREROUTING (policy ACCEPT 146M packets, 19G bytes)
pkts bytes target prot opt in out source destination
33M 4144M CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 CT notrack
33M 2805M CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 CT notrack
Chain OUTPUT (policy ACCEPT 73M packets, 8311M bytes)
pkts bytes target prot opt in out source destination
10785 882K CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 CT notrack
0 0 CT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 CT notrack
That left the count hovering around 13000 or so, and nf_conntrack_max is set to 65535. However I still keep getting the dropped packet messages. Most of the rest of the packets are UDP, and I set the nf_conntrack_udp_timeout as low as 1 second, leaving the nf_conntrack_count around 1000. However I still get the dropped packed message.
From here, if I raise the max, it will stop the dropped packet messages, however I don't see why that is necessary.
I am running docker, and there is an elasticsearch container (this problem seems to happen on whichever node is running elasticsearch). Not sure if it is relevant, but the node has 48 cores.
$ uname -a
Linux qtausc-pphd0128 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
So why is it dropping packets when the count is far less than the max?
Best Answer
I had the same issue a while back on a Squid system.
One of the most effective way I found to reduce the size of the conntrack was to reduce the default TCP timeout in the kernel.
The
net.netfilter.nf_conntrack_tcp_timeout_established
is set to 432000 by default. That's right...that's 5 days.To set the value you can issue the following command;
And if you want that change to be persistent you need to add the line to
/etc/sysctl.conf
.After reducing that value to 600 the conntrack count was steadily going down over a couple of days.
I used
sysctl net.netfilter.nf_conntrack_max
andsysctl net.netfilter.nf_conntrack_count
in order to get the values.