Linux – Top shows load of 2.5: how can I find out which processes are waiting for the CPU

linuxtop

I have a linux box hosting a pretty low-traffic site. It's an Amazon EC2 "small" instance, running Ubuntu 10.04. When I run "top" on it, I usually see a load of between 2 and 3, which in my experience is pretty high. However, not much seems to really be happening on the machine — the CPU is almost always idle, and by watching the apache2 access.log file, I can see that not a lot of requests are coming through. How can I figure out what processes are waiting for the CPU, so that I can try to understand why the load metric is so high?

Best Answer

Using top you can see which threads are running and which ones are sleeping. This should allow you to at least know what is draining your resources if we're talking about a CPU bottleneck.

[xxx@absynthe proc]$ top -H

Should bring up a screen like this: 

top - 17:54:38 up 37 min,  2 users,  load average: 0.03, 0.06, 0.07
Tasks: 338 total,   2 running, 336 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.1%us,  2.3%sy,  0.0%ni, 92.1%id,  1.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3852932k total,  1596468k used,  2256464k free,    47108k buffers
Swap:  5963768k total,        0k used,  5963768k free,   681728k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                           
 1853 root      20   0  196m  38m  15m S  9.7  1.0   1:57.58 Xorg                                                                                                                                              
 2186 xxx       -6   0  564m 8828 7216 S  3.9  0.2   0:43.69 pulseaudio                                                                                                                                        
 2611 xxx       20   0 1095m 235m  27m S  3.9  6.3   2:29.52 firefox                                                                                                                                           
 2179 xxx        9 -11  564m 8828 7216 S  1.9  0.2   0:38.34 pulseaudio                                                                                                                                        
 2671 xxx       20   0 1087m  43m  18m S  1.9  1.2   0:06.06 plugin-containe                                                                                                                                   
 2820 xxx       20   0 1275m  67m  23m S  1.9  1.8   0:13.13 souphttpsrc13:s                                                                                                                                   
 2824 xxx       20   0  315m  13m 9492 S  1.9  0.4   0:02.35 gnome-terminal                                                                                                                                    
 3114 xxx       20   0 15088 1300  820 R  1.9  0.0   0:00.02 top                                                                                                                                               
    1 root      20   0 19236 1440 1152 S  0.0  0.0   0:01.07 init                                                                                                                                              
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                          
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                                                                                       
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.17 ksoftirqd/0                                                                                                                                       
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                                                                                        
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1

The column named S (that is the 8th one from the right) shows S for sleeping threads and R for running threads. You can choose the sorting order by pressing F (capital) and choosing the Process Status field. You can then reverse the sort order by pressing R (capital) so you can see the running threads first.

If your problem is not a CPU bottleneck I'll need some more info to assist you. Maybe you could post your top like I did on the above example.

Hope this helps.

EDIT:

If you want to know if you're experiencing some kind of I/O bottleneck you can issue the following command: vmstat -s and look for the IO-wait CPU ticks. If, running the command with a couple of seconds interval, the value goes up a lot you might be experiencing an I/O bottleneck. In that case you might be better off using iotop to see what processes are using more I/O resources.

Related Topic