Show full process name in top

The CPU can and will be used for other processes, provided there is at least one process that is ready to receive CPU time. There's the rub - you can have an I/O-bound system with every process waiting for I/O to complete, and as there is nothing waiting for CPU time, there is no reason to schedule (and utilize) CPU for anything other than the kernel's activities...hence the term, I/O wait.

Try running vmstat 1 and see if there are numbers greater than 0 in the "b" column (2nd column) on a regular basis. If so, you're probably I/O-bound. Seeing it occasionally isn't a big deal, seeing it all the time with numbers in the 2-3 range is tolerable but not desireable, and seeing more than 5+ means you're probably way too busy (although that depends on how much I/O your system can accomodate, so it can be more or less, depending). The "b" means "processes blocked", as in, "the number of processes scheduled to run, but were blocked, pending the completion of an I/O".

Follow-up:

There is a known bug with heavy I/O and the newer schedulers on the 2.6 series of kernels. Try changing your scheduler to see if it has an impact.

semaphores were maxed out. i dont know why apache has allocated itself ~115 semaphores but upping the value fixed all the problems - apache no longer has to IOWAIT for another semaphore, so the system doesnt hang.

echo "kernel.sem = 512 32000 100 512" >> /etc/sysctl.conf; sysctl -p;