I have a server running Xen and a couple of VMs. I'm trying to set up a RAID array that's dedicated to one of the VMs in particular, which will be used for various storage-intensive purposes. Currently, I'm having a very odd performance drop when writing from the domU, which is a Debian PV guest with plenty of vcpus and memory.
Right now, the setup is that I have three 3TB WD Red hard drives arranged in a (software) RAID 5 array on the dom0. Currently, the dom0 is exposing the /dev/mdX block as /dev/xvdb on the virtual machine (the VM has /dev/xvda from a LVM volume). The relevant bit of the xen config:
disk = [ <LVM stuff for xvda>, 'phy:/dev/md0,xvdb,w' ]
The /dev/md0 has a ext4 filesystem on it, with the noatime, nodiratime options in use. When I do a speed test on the filesystem from the domU, I get something like this:
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 16.2694 s, 66.0 MB/s
# dd if=some_other_noncached_file of=/dev/null bs=1M
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 26.379 s, 163 MB/s
However, in dom0, I get:
# dd if=/dev/zero of=a_file_somewhere bs=1M count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.80677 s, 158 MB/s
… which, clearly, is a very large speed difference. While 66 MB/s should be good enough for now, I'd really appreciate it if anyone can come up with an explanation for why I'm losing 60% of my write performance? I'd be expectant of 10% or so, but not 60%.
It's not dom0 resource starvation, because I've been giving it significantly more resources than it should require and the problem still occurs. It's not domU resource starvation, either, and I've pinned CPUs to stay on the same NUMA node, and the same thing results.
Here's some maybe-relevant things from xl info:
host : <hostname>
release : 3.16.0-4-amd64
version : #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
machine : x86_64
nr_cpus : 24
max_cpu_id : 63
nr_nodes : 2
cores_per_socket : 6
threads_per_core : 2
cpu_mhz : 2660
hw_caps : bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps : hvm
total_memory : 24573
free_memory : 12011
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 4
xen_extra : .1
xen_version : 4.4.1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=8192M dom0_max_vcpus=8 dom0_vcpus_pin
cc_compiler : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by : ultrotter
cc_compile_domain : debian.org
cc_compile_date : Thu Jun 11 18:24:17 EEST 2015
xend_config_format : 4
If there's anything else that might be useful, please let me know and I'll update the question.
Thanks!
Best Answer
I've seen near native disk performance with PV guests, with the MD work being done at both the dom0 and domU levels.
If you can, try setting up guest as paravirtualized (PV) instead of a hardware virtual machine (HVM).
Also, for either PV or HVM, try assigning the whole drives to the guest and having it handle the RAID stuff (MD).
See what works best for you.