Linux – High %system and %si cpu time on Linux

linux

On top and iostat output it seems there is some problem with system and softirq items.

1- Do you think this only caused by ethernet interrupts?

2- I am running netfilter with 60 Mbps traffic on that server. How can I see how much of %system is used by netfilter and other kernel modules?

3- Time to time, system hangs with OOM condition. However at that moment system has lots of free memory according to sar output. Will higher %system and %idle cause blocking in reaching memory by kernel?

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          34.17    0.00   50.25    0.00    0.00   15.58


Cpu0  : 37.5%us, 18.8%sy,  0.0%ni,  6.2%id,  0.0%wa,  0.0%hi, 37.5%si,  0.0%st
Cpu1  : 33.3%us, 44.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 22.2%si,  0.0%st


cat /proc/interrupts                              Thu Jul 28 11:55:11 2011

           CPU0       CPU1
  0:  136049409      43332    IO-APIC-edge  timer
  1:          1          1    IO-APIC-edge  i8042
  4:          5          7    IO-APIC-edge  serial
  7:          0          0    IO-APIC-edge  parport0
  8:          2          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:          2          2    IO-APIC-edge  i8042
 50:          0          0   IO-APIC-level  uhci_hcd:usb4
 58:          0          0   IO-APIC-level  uhci_hcd:usb7
 66:      80860    4360213         PCI-MSI  ahci
 74:    1698104  274488209   IO-APIC-level  eth1
 98:  157341956    3131716         PCI-MSI  eth4
106:         92         95   IO-APIC-level  HDA Intel
169:          0          0   IO-APIC-level  uhci_hcd:usb3
225:          0          0   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb

Best Answer

1- What do you mean when you said "ethernet interrupts"?

2- A nearly exact way:

$ echo "$(cat /proc/`pidof iptables`/stat | awk '{ print $15 }') * 100 / $(cat /proc/stat | awk '/cpu / { print $4 }')" | bc -l

Take a look at man proc for more details.

3- Find out which process is eating your RAM with:

$ ps -eo pmem,pid,comm | sort -k1rn | head

or atop, htop, ...