High %soft CPU usage during high network load on ixgbe

networkingperformance

We have an Intel 82598EB 10GBE NIC. The network load is composed of lot of small sized UDP packets. To simulate a DDoS, from another server we pushed ~100k pps small UDP packet (just some Mbit of bandwidth). When we do, %soft CPU usage goes to 100% making the server unusable and dropping packets.

Kernel is 3.8.5, ixgbe 3.13.10
We have already made some tunings to lower interrupts rate (rx-usecs: 300) but didn't solve the problem. Is there a way to check why softirqs use so much CPU even if interrupts rate is not high?

Thank you!

Best Answer

I know this is old; but I'm mainly replying for users who come here with similar issues.

There are a few ways to check interrupts:

  • cat /proc/interrupts records interrupts per CPU and IO device.
  • grep cpu /proc/stat column 6 and 7 contain irq and softirq stats.
  • top and htop; observe si and if you change htop to advanced display mode for CPUs you can see si and hi.
  • To get historical data install sysstat; to get specific interrupt data see sar -I XALL.
  • You can also use perf.

Since this is most likely a driver issue (or the NIC is failing) I'd check for any overruns or errors on the interface; if there are none see if there is an update to the driver.

Related Topic