I have a decent CentOS 6.5 dedicated host (CentOS 6.5/E3-1230 3.2Ghz Quad Core + HT/16GB/Software Raid 1 SATA II/WD2503ABYX/ext4) with the default CentOS kernel and "elevator=deadline" in grub.
I/O write operations cause a huge spike in CPU usage. Reads are working fine. For example,
dd if=/dev/zero of=test bs=1048576 count=2048
causes the host's CPU utilization to shoot above 3 or 4. Under normal operation it remains under 0.40, but when there is some more intense I/O operation everything grinds to a halt.
mpstat 1
during these dd
tests shows io wait at 20-25%.
Here's the disk layout:
Disk /dev/sda: 251.1 GB, 251059544064 bytes
255 heads, 63 sectors/track, 30522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6673
Device Boot Start End Blocks Id System
/dev/sda1 * 1 26 204800 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 26 548 4194304 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sda3 548 30523 240775168 fd Linux raid autodetect
Disk /dev/sdb: 251.1 GB, 251059544064 bytes
255 heads, 63 sectors/track, 30522 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00095c99
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 26 204800 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2 26 548 4194304 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary.
/dev/sdb3 548 30523 240775168 fd Linux raid autodetect
Disk /dev/md2: 246.6 GB, 246552588288 bytes
2 heads, 4 sectors/track, 60193503 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md1: 4293 MB, 4293910528 bytes
2 heads, 4 sectors/track, 1048318 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vg_main-LogVol00: 246.5 GB, 246549577728 bytes
255 heads, 63 sectors/track, 29974 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md0: 209 MB, 209702912 bytes
2 heads, 4 sectors/track, 51197 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
The problem (high CPU usage) started happening sometime late december last year, which leads me to believe it is software related (the disk susbsystem has been checked by the folks at the DC).
Which tests should I run next to try to isolate the problem?
PS: I'm not looking for performance maximization tips. The server is underutilized. I'm just looking to reduce the CPU load during disk writes.
UPDATE: question reworked to better describe the problem.
UPDATE: FOUND A SOLUTION I finally discovered what the problem was when I came across this post.
root> modprobe vhost_net
root> echo vhost_net > /etc/modules
For some reason, the virtio interfaces were not loading the driver before. All is good now.
Best Answer
On CentOS, the
dirty_ratio
is set to 20%.What this means is that writing a file out doing
Actually writes the data into memory as writeback (up to 3.2GB) and does not actually write it out to disk.
Its slower (but not a realistic performance benchmark) on the VM because you've probably assigned a much lower memory assignment to the VM itself (lets say 2G) and this leads to
dirty_writeback
only providing ~400MB of writeback before it will force the contents to disk.If you run that command, then run
sync
, you'll notice sync takes a long time to return.You need to run your command doing the following instead to get a better idea of your actual throughput.