Linux – High load average (over 2.0) on an idle machine

linuxload-average

I've this server which is basically idle, and yet has a high load average.

  • Hardware : 4 processor PowerPC
  • Over 4GB of free RAM
  • Top says CPUs are 99.9% idle
  • Virtually no disk I/O
  • Debian Squeeze, default install except I'm using ext4

Here's the output of a few commands :

uname -a

Linux box 2.6.32-5-powerpc64 #1 SMP Tue Mar 8 02:01:42 UTC 2011 ppc64 GNU/Linux

top

top - 14:08:57 up  1:58,  1 user,  load average: 2.68, 2.45, 2.29
Tasks: 105 total,   1 running, 104 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4987256k total,  4965484k used,    21772k free,    16540k buffers
Swap: 24414028k total,        0k used, 24414028k free,  4781172k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2606 myself    20   0  3276 1340 1076 R    0  0.0   0:00.62 top
    1 root      20   0  2560  844  740 S    0  0.0   0:00.65 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0  

uptime

 14:09:23 up  1:58,  1 user,  load average: 2.54, 2.43, 2.28

iostat -d 2 -m

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sdb               0.00         0.00         0.00          0          0
sda               1.50         0.00         0.02          0          0

free -m

             total       used       free     shared    buffers     cached
Mem:          4870       4853         17          0         16       4669
-/+ buffers/cache:        167       4702
Swap:        23841          0      23841

ps axf

  PID TTY      STAT   TIME COMMAND
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:00  \_ [migration/0]
    4 ?        S      0:00  \_ [ksoftirqd/0]
    5 ?        S      0:00  \_ [watchdog/0]
    6 ?        S      0:00  \_ [migration/1]
    7 ?        S      0:00  \_ [ksoftirqd/1]
    8 ?        S      0:00  \_ [watchdog/1]
    9 ?        S      0:00  \_ [migration/2]
   10 ?        S      0:00  \_ [ksoftirqd/2]
   11 ?        S      0:00  \_ [watchdog/2]
   12 ?        S      0:00  \_ [migration/3]
   13 ?        S      0:00  \_ [ksoftirqd/3]
   14 ?        S      0:00  \_ [watchdog/3]
   15 ?        S      0:00  \_ [events/0]
   16 ?        S      0:00  \_ [events/1]
   17 ?        S      0:00  \_ [events/2]
   18 ?        S      0:00  \_ [events/3]
   19 ?        S      0:00  \_ [cpuset]
   20 ?        S      0:00  \_ [khelper]
   21 ?        S      0:00  \_ [netns]
   22 ?        S      0:00  \_ [async/mgr]
   23 ?        S      0:00  \_ [pm]
   24 ?        S      0:00  \_ [sync_supers]
   25 ?        S      0:00  \_ [bdi-default]
   26 ?        S      0:00  \_ [kintegrityd/0]
   27 ?        S      0:00  \_ [kintegrityd/1]
   28 ?        S      0:00  \_ [kintegrityd/2]
   29 ?        S      0:00  \_ [kintegrityd/3]
   30 ?        S      0:00  \_ [kblockd/0]
   31 ?        S      0:00  \_ [kblockd/1]
   32 ?        S      0:00  \_ [kblockd/2]
   33 ?        S      0:00  \_ [kblockd/3]
   38 ?        S      0:00  \_ [khungtaskd]
   39 ?        S      0:04  \_ [kswapd0]
   40 ?        SN     0:00  \_ [ksmd]
   41 ?        S      0:00  \_ [aio/0]
   42 ?        S      0:00  \_ [aio/1]
   43 ?        S      0:00  \_ [aio/2]
   44 ?        S      0:00  \_ [aio/3]
   45 ?        S      0:00  \_ [crypto/0]
   46 ?        S      0:00  \_ [crypto/1]
   47 ?        S      0:00  \_ [crypto/2]
   48 ?        S      0:00  \_ [crypto/3]
  134 ?        S      0:00  \_ [ksuspend_usbd]
  135 ?        S      0:00  \_ [kmmcd]
  137 ?        S      0:00  \_ [ata/0]
  138 ?        S      0:00  \_ [ata/1]
  139 ?        S      0:00  \_ [ata/2]
  140 ?        S      0:00  \_ [ata/3]
  141 ?        S      0:00  \_ [ata_aux]
  142 ?        S      0:00  \_ [scsi_eh_0]
  143 ?        S      0:00  \_ [scsi_eh_1]
  144 ?        S      0:00  \_ [scsi_eh_2]
  145 ?        S      0:00  \_ [scsi_eh_3]
  150 ?        S      0:00  \_ [khubd]
  174 ?        S      0:00  \_ [usbhid_resumer]
  227 ?        D      0:00  \_ [kwindfarm]
  239 ?        S      0:00  \_ [jbd2/sda3-8]
  240 ?        S      0:00  \_ [ext4-dio-unwrit]
  241 ?        S      0:00  \_ [ext4-dio-unwrit]
  242 ?        S      0:00  \_ [ext4-dio-unwrit]
  243 ?        S      0:00  \_ [ext4-dio-unwrit]
  424 ?        S      0:00  \_ [nouveau/0]
  425 ?        S      0:00  \_ [nouveau/1]
  426 ?        S      0:00  \_ [nouveau/2]
  427 ?        S      0:00  \_ [nouveau/3]
  459 ?        S      0:00  \_ [phy0]
  474 ?        S      0:00  \_ [flush-8:0]
  493 ?        S      0:00  \_ [ttm_swap]
  588 ?        S      0:00  \_ [bluetooth]
  635 ?        S      0:00  \_ [firewire_sbp2]
  693 ?        S      0:00  \_ [jbd2/sda5-8]
  694 ?        S      0:00  \_ [ext4-dio-unwrit]
  695 ?        S      0:00  \_ [ext4-dio-unwrit]
  696 ?        S      0:00  \_ [ext4-dio-unwrit]
  697 ?        S      0:00  \_ [ext4-dio-unwrit]
 1694 ?        S      0:02  \_ [jbd2/sdb1-8]
 1695 ?        S      0:00  \_ [ext4-dio-unwrit]
 1696 ?        S      0:00  \_ [ext4-dio-unwrit]
 1697 ?        S      0:00  \_ [ext4-dio-unwrit]
 1698 ?        S      0:00  \_ [ext4-dio-unwrit]
    1 ?        Ss     0:00 init [2]  
  303 ?        S<s    0:00 udevd --daemon
  368 ?        S<     0:00  \_ udevd --daemon
 1385 ?        S<     0:00  \_ udevd --daemon
  929 ?        Sl     0:00 /usr/sbin/rsyslogd -c4
  998 ?        Ss     0:00 /usr/sbin/atd
 1042 ?        Ss     0:00 /usr/sbin/cron
 1255 ?        Ss     0:00 /usr/sbin/exim4 -bd -q30m
 1286 tty2     Ss+    0:00 /sbin/getty 38400 tty2
 1287 tty3     Ss+    0:00 /sbin/getty 38400 tty3
 1288 tty4     Ss+    0:00 /sbin/getty 38400 tty4
 1289 tty5     Ss+    0:00 /sbin/getty 38400 tty5
 1290 tty6     Ss+    0:00 /sbin/getty 38400 tty6
 1300 ?        Ss     0:00 dhclient -v -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
 1384 tty1     Ss+    0:00 /sbin/getty 38400 tty1
 2113 ?        Ss     0:00 /usr/sbin/apache2 -k start
 2116 ?        S      0:00  \_ /usr/sbin/apache2 -k start
 2118 ?        Sl     0:00  \_ /usr/sbin/apache2 -k start
 2119 ?        Sl     0:00  \_ /usr/sbin/apache2 -k start
 2577 ?        Ss     0:00 /usr/sbin/sshd

Best Answer

Try to upgrade/downgrade your kernel. Are several problems with the sheduler on different kernels:

  1. http://lkml.indiana.edu/hypermail//linux/kernel/1010.1/02354.html
  2. http://www.gossamer-threads.com/lists/linux/kernel/1169420