KVM bad IO sync performance in Linux guest

iokvm-virtualizationlibvirtperformancevirtualization

I'm setting up virtualized Ubuntu 15.10 guest inside Ubuntu 15.04 server using libvirt and KVM.

Storage in host system:

# mount | grep ext4
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sdb1 on /media/hdd type ext4 (rw,relatime,data=ordered)

/dev/sda is an SSD and /dev/sdb is a HDD.

The initial configuration was done using virt-manager.

Problem:

Overall performance of guest system seems to be good, however some operations, noticeably installing packages using apt-get (and therefore guest system installation) are very slow.

Guest disk image is in raw format, and I've tried placing it on both SSD and HDD as a file, and also as a separate partition on HDD. Different locations make some difference on performance, however installing some package using apt-get is still two or three times slower than on host system.

I've also tried various qemu disk cache options, and using unsafe cache seems to eliminate the problem completely. But, any other cache setting, including none and writeback shows bad performance. Of course, using unsafe doesn't look like a real solution.

Here is an excerpt from the guest xml config:

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source dev='/dev/sdb2'/>
  <target dev='vda' bus='virtio'/>
  <boot order='2'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</disk>

When using image file instead of a partition that becomes:

<disk type='file' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native'/>
  <source file='/media/hdd/dsi/vm/ubuntu4.bin'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <boot order='2'/>
  <alias name='virtio-disk0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>

While trying to fix the issue, I've noticed that the first partition on the HDD was not properly aligned, so I re-created the partition with correct alignment (which for this HDD is 4096). This also hasn't helped much.

I've also tried setting elevator=noop for the guest, to no avail.

When I run similar guest system on Windows PC using VirtualBox, I don't experience such performance degradation, so I hope there's a way to run the guest with the same performance on KVM also.

Best Answer

Try

cache=directsync

Or

cache=writethrough

Also

vm.swappiness=1
vm.vfs_cache_pressure=75 (play around with this)

Use deadline instead of cfq/noop (poor performers on ssds and raid I've found).

Ignore what people say about cache=none or writeback being the best. It not always that simple.