Linux – High iowait despite small amount of write/read

iowaitlinuxoracle-11g

I'm not sure how to interpret this or the next steps, but why is IOWait so high when I have not written/read much from any devices?

Background: I am a developer trying to find why the DB seems to be slow on a particular environment.

iostat -tm 60 15 Time: 05:17:01 PM avg-cpu: %user %nice %system %iowait %steal %idle 2.43 0.00 2.28 16.50 0.00 78.78 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn cciss/c0d0 2.14 0.00 0.03 70614 859153 cciss/c0d0p1 0.00 0.00 0.00 41 2 cciss/c0d0p2 2.14 0.00 0.03 70572 859151 dm-0 2.51 0.00 0.01 19375 296915 dm-1 1.69 0.00 0.02 38433 548545 dm-2 0.22 0.00 0.00 12764 13689

Time: 05:18:01 PM avg-cpu: %user %nice %system %iowait %steal %idle 4.84 0.00 5.21 54.52 0.00 35.43 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn cciss/c0d0 1.78 0.00 0.02 0 1 cciss/c0d0p1 0.00 0.00 0.00 0 0 cciss/c0d0p2 1.78 0.00 0.02 0 1 dm-0 2.47 0.00 0.01 0 0 dm-1 1.22 0.00 0.01 0 0 dm-2 0.00 0.00 0.00 0 0

Time: 05:19:01 PM avg-cpu: %user %nice %system %iowait %steal %idle 5.71 0.00 5.24 58.33 0.00 30.72 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn cciss/c0d0 1.68 0.00 0.02 0 1 cciss/c0d0p1 0.00 0.00 0.00 0 0 cciss/c0d0p2 1.68 0.00 0.02 0 1 dm-0 1.88 0.00 0.01 0 0 dm-1 1.18 0.00 0.01 0 0 dm-2 0.00 0.00 0.00 0 0

Best Answer

You can run awr reports with the following. Oracle will tell you what it is waiting on in under "Top 5 Timed Events"

-- run awr (timed_statistics=TYPICAL and control_management_pack_access='DIAGNOSTIC+TUNING')
connect / as sysdba
@$ORACLE_HOME/rdbms/admin/awrrpt.sql

Related Solutions

Samba – Resolving high iowait on Samba server

Check the io scheduler (called the io elevator in the kernel documentation) assigned to that volume.

$ cat /sys/block/sda/queue/scheduler 
noop anticipatory [deadline] cfq

For most RedHat and Fedora distros the default is the CFQ scheduler. IMHO this is not the best for an iobound server process. I recommend the deadline scheduler

$ echo "deadline" > /sys/block/sda/queue/scheduler

to change this, or add

elevator=deadline

to your boot arguments to make this permanent.

High iowait times can also be caused by aggressive readahead, which may be unnecessary for your workload. The default for RedHat and Fedora systems is 128k

$ cat /sys/block/sda/queue/read_ahead_kb 
128

Experiment with echo'ing lower values to that file and see if that decreases your iowait times.

Also, check the underlying disk or array. If it is degraded, or rebuilding, your iowait times will skyrocket as the underlying disk subsystem steals io bandwidth while rebuilding itself

Ubuntu 10.10 Server Random System Lockups – How to Resolve

I just reported a bug to Launchpad about the very same issue, five days ago. It is also an Intel Core i7-930, in an Intel DX58SO mainboard:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/665796

Most certainly yours is the same problem, your description matches too much with my problem. Check if your system timer interrupt freezes during these lockups. Keep this running in a terminal:

watch -d grep timer /proc/interrupts

You will need an external source of interrupts during the lockup in order to make the process scheduler operate and see the command executing. Pinging your computer from another machine (with a small interval) will help.

If the timer interrupt stops incrementing during the lockup, then it is the same problem. Please, add your system information to the above bug report so that we get attention from the Ubuntu developers:

ubuntu-bug -u 665796 -p linux

Related Topic