Linux – Top showing 64% idle on first screen or batch run, while there is no idle time at all

Using top you can see which threads are running and which ones are sleeping. This should allow you to at least know what is draining your resources if we're talking about a CPU bottleneck.

[xxx@absynthe proc]$ top -H

Should bring up a screen like this:

top - 17:54:38 up 37 min,  2 users,  load average: 0.03, 0.06, 0.07
Tasks: 338 total,   2 running, 336 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.1%us,  2.3%sy,  0.0%ni, 92.1%id,  1.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3852932k total,  1596468k used,  2256464k free,    47108k buffers
Swap:  5963768k total,        0k used,  5963768k free,   681728k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                           
 1853 root      20   0  196m  38m  15m S  9.7  1.0   1:57.58 Xorg                                                                                                                                              
 2186 xxx       -6   0  564m 8828 7216 S  3.9  0.2   0:43.69 pulseaudio                                                                                                                                        
 2611 xxx       20   0 1095m 235m  27m S  3.9  6.3   2:29.52 firefox                                                                                                                                           
 2179 xxx        9 -11  564m 8828 7216 S  1.9  0.2   0:38.34 pulseaudio                                                                                                                                        
 2671 xxx       20   0 1087m  43m  18m S  1.9  1.2   0:06.06 plugin-containe                                                                                                                                   
 2820 xxx       20   0 1275m  67m  23m S  1.9  1.8   0:13.13 souphttpsrc13:s                                                                                                                                   
 2824 xxx       20   0  315m  13m 9492 S  1.9  0.4   0:02.35 gnome-terminal                                                                                                                                    
 3114 xxx       20   0 15088 1300  820 R  1.9  0.0   0:00.02 top                                                                                                                                               
    1 root      20   0 19236 1440 1152 S  0.0  0.0   0:01.07 init                                                                                                                                              
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                          
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                                                       
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.17 ksoftirqd/0                                                                                                                                       
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                                                                                        
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                                                                                                       

The column named S (that is the 8th one from the right) shows S for sleeping threads and R for running threads. You can choose the sorting order by pressing F (capital) and choosing the Process Status field. You can then reverse the sort order by pressing R (capital) so you can see the running threads first.

If your problem is not a CPU bottleneck I'll need some more info to assist you. Maybe you could post your top like I did on the above example.

Hope this helps.

EDIT:

If you want to know if you're experiencing some kind of I/O bottleneck you can issue the following command: vmstat -s and look for the IO-wait CPU ticks. If, running the command with a couple of seconds interval, the value goes up a lot you might be experiencing an I/O bottleneck. In that case you might be better off using iotop to see what processes are using more I/O resources.

I would recommend searching for anything non in the S sleeping state. It's possible you've got zombie processes which can get counted as something running, despite not really doing anything. ps -eo pid,user,state,cmd | awk '$3 !~ /S/ {print $0}' This will show any non-sleeping processes. (Running, waiting on IO, zombied, etc)

It's worth noting that your load average isn't terribly alarming. Assuming you have more than two cores on the box, there's no doubt plenty of CPU power to go around. But obviously still worth looking into if you don't expect 1-2 processes running at any given time.


--Christopher Karel