Linux – What creates CPU I/O wait but no disk operations

You can get a pretty good measure of this using the iostat tool.

% iostat -dx /dev/sda 5

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.78    11.03    1.19    2.82    72.98   111.07    45.80     0.13   32.78   1.60   0.64

The disk utilisation is listed in the last column. This is defined as

Percentage of CPU time during which I/O requests were issued to the device (band-width utilization for the device). Device saturation occurs when this value is close to 100%.

As noted load average is related to the number of waiting processes in the run queue. If each of those processes has very little work to do and frees the processor quickly you can handle much larger load averages than the common 1 per CPU rule of thumb.

Mail is pretty much the perfect example of this, each process needs CPU to send a message but very, very little. I've seen mail systems running sendmail at a load average in the 25 to 35 range, and the system is still interactive and working fine.

Mark