Java – High load without explanation

javaweb-applications

I have a very high load on my machine and don't know what is responsible or how to find out.

On the machine runs a jboss appserver and mysql. Here is a top from the user at peak time:

top - 16:23:01 up 101 days,  6:50,  1 user,  load average: 23.42, 21.53, 24.73
Tasks:   9 total,   1 running,   8 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.2%us,  1.6%sy,  0.0%ni, 80.4%id,  0.1%wa,  0.1%hi,  0.7%si,  0.0%st
Mem:  16440784k total, 16263720k used,   177064k free,   151916k buffers
Swap: 16780872k total,    30428k used, 16750444k free,  8963648k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27344 b         40   0 16.0g 6.5g  14m S  169 41.7   1184:09 java
 6047 b         40   0 11484 1232 1228 S    0  0.0   0:00.01 mysqld_safe
 6192 b         40   0  604m 182m 4696 S    0  1.1  93:30.40 mysqld
 7948 b         40   0 84036 1968 1176 S    0  0.0   0:00.07 sshd
 7949 b         40   0 14004 2900 1608 S    0  0.0   0:00.03 bash
 7975 b         40   0  8604 1044  840 S    0  0.0   0:00.44 top

The CPU usage of the java process is normal. The peaks only show up when i deployed a certain web application. Could the resulting network traffic boost the load in such way that i don't see it in top?

Best Answer

So load average is actually quite complicated, but my understanding that it basically is what is waiting in the run queue. So my guess is that you may have things waiting on IO. Here is a nice stolen snippet to see what is waiting:

ps -eo stat,pid,user,command | egrep "^STAT|^D|^R"

D : Uninterruptible sleep (usually IO)
R : Running or runnable (on run queue)

As pointed out, iostat works well as well to see if it likely disk.