Kernel: dst cache overflow

kernelnetworkingoverflow

I'm under continuous floods at time that are generally of lower magnitudes. I've observed my system often to become totally unresponsive on its network while I observed this in the logs,

kernel: [455951.513204] __ratelimit: 1771 callbacks suppressed
kernel: [455951.513207] dst cache overflow

I've researched a lot on this but did not find any suitable answer. I'm running the kernel version 2.6.32-5-amd64 by the way on a Debian 6 system.

Best Answer

This page (http://bluecoat.force.com/knowledgebase/articles/Solution/CB-IPdestinationcacheoverflowdstcacheoverflowipdstcachemessage) has instructions for increasing the cache associated with this message - quoted below.

However, if you are "under continuous floods" increasing this cache may only make the flooding problem worse, and it is probably better to address the problem before it gets to your system. Perhaps by dropping offending traffic at your router/firewall.

Use the following procedure to assess the situation and change the size of the dest cache. All of the following commands assume that you are connected to the VAP console..

Check the current situation: cat /proc/slabinfo |grep ip_dst_cache

And settings: cat /proc/sys/net/ipv4/route/max_size

Set a new maximum value (ie. 2621440) and verify that it was accepted: echo 2621440 > /proc/sys/net/ipv4/route/max_size; cat /proc/sys/net/ipv4/route/max_size

Check the current situation again: cat /proc/slabinfo | grep ip_dst_cache

After a while, the CPU load should go down.

Another issue that a customer might experience is that the Firewall connections table is full. When this happens, more memory is required from the system.

If the firewall is not deployed in the core part of network this issue shouldn't occur. If the customer still sees this condition, it might be caused by somebody trying to spoof IP addresses in the internal network by doing a kind of nmap scan with spoofed IP addresses or something like that. For a typical data center in front of servers and/or a perimeter firewall it shouldn't be observed. The cause might be some kind of DoS/DDoS attack. Regardless of the source of the issue, the procedure above will solve the issue.

All sysctl parameters are loaded at boot time through /etc/init.d/network script. The command is:

sysctl -e -p /etc/sysctl.conf

This script starts before the Check Point process and that is why the changes don't survive a reboot.

When Check Point is installed this value is configured to 524288 when firewall starts by fwstart script. So, even though we change the parameter in the /etc/sysctl.conf file, and Linux configures it at boot time, when the firewall starts, this value is changed again. Then, if we just stop (cpstop) and start (cpstart) the firewall, these values would be changed again.

Check Point is changing this value - $ cd $FWDIR/bin $ grep -n max_size fwstart echo 524288 > /proc/sys/net/ipv4/route/max_size

To make sure the kernel will have the correct value after a reboot or firewall restart do the following:

  1. Configure files below to reflect the right value

    • /etc/sysctl.conf

  2. Disable line in fwstart script ($FWDIR/bin/fwstart)

    echo 524288 > /proc/sys/net/ipv4/route/max_size

NOTE: After applying the Check Point HFA or upgrading Check Point, the fwstart script might be overwritten.

To get real time changes, use this command:

$ sysctl -w net.ipv4.route.max_size = 2097152

Related Topic