For a long period of time, i'm experiencing performance issues on CentOS servers that are used for KVM virtualization.
I'm running software RAID-10 with 4 disks (Seagate ST2000DM001, 2TB, 7200rpm SATA-600)
These KVM VM's are running off their own LV.
Once there are a few VM's installed, the performance keeps dropping with every VM that's added.
Right now, it takes about 3 hours to install Debian 7 minimal. On an empty box with no VM's, this will take just a few minutes max.
The weird thing is that even with 37 VM's, there is pretty much no disk I/O. But every I/O that needs to be done at that moment, brings the "dm" device regarding that specific LV volume to 100%.
For some reason it looks like the "dm-xx" devices are not capable of using all the disks.
Down below is an example of what I mean. The sd[a/b/c/d] are usually about 40 to 60% loaded during this problem, but one or more dm devices are having high load without any serious I/O.
Notice that the %util for specific dm-0's that are in use by VM's.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 10.00 0.00 5.00 36.00 256.00 424.00 33.17 0.86 20.73 11.02 45.20
sdd 0.00 1.00 0.00 48.00 0.00 389.00 16.21 0.85 17.71 10.90 52.30
sda 0.00 0.00 2.00 39.00 32.00 161.00 9.41 0.77 18.63 11.44 46.90
sdc 0.00 1.00 0.00 50.00 0.00 764.00 30.56 1.59 47.08 13.90 69.50
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md3 0.00 0.00 18.00 68.00 288.00 813.00 25.60 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 6.00 0.00 40.00 13.33 0.31 51.50 51.17 30.70
dm-1 0.00 0.00 0.00 7.00 0.00 72.00 20.57 0.74 49.43 99.14 69.40
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 16.00 0.00 256.00 0.00 32.00 0.96 59.69 8.38 13.40
dm-4 0.00 0.00 0.00 8.00 0.00 68.00 17.00 0.28 34.62 31.25 25.00
dm-5 0.00 0.00 0.00 4.00 0.00 156.00 78.00 0.05 12.00 12.00 4.80
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-7 0.00 0.00 0.00 4.00 0.00 20.00 10.00 0.08 19.00 6.75 2.70
dm-8 0.00 0.00 0.00 3.00 0.00 4.00 2.67 0.54 258.33 154.00 46.20
dm-9 0.00 0.00 0.00 5.00 0.00 148.00 59.20 0.10 19.40 19.40 9.70
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 2.00 0.00 32.00 0.00 32.00 0.03 12.50 6.50 1.30
dm-19 0.00 0.00 0.00 2.00 0.00 12.00 12.00 0.03 13.00 6.50 1.30
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.11 294.00 110.00 11.00
dm-28 0.00 0.00 0.00 3.00 0.00 16.00 10.67 0.10 50.33 25.00 7.50
dm-29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-31 0.00 0.00 0.00 4.00 0.00 12.00 6.00 0.56 140.25 140.25 56.10
dm-32 0.00 0.00 0.00 6.00 0.00 32.00 10.67 0.76 125.83 125.83 75.50
dm-33 0.00 0.00 0.00 4.00 0.00 12.00 6.00 0.21 53.00 53.00 21.20
dm-34 0.00 0.00 0.00 3.00 0.00 4.00 2.67 0.40 172.33 133.33 40.00
dm-35 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-37 0.00 0.00 0.00 9.00 0.00 217.00 48.22 1.09 135.33 111.00 99.90
=================================
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 0.00 0.00 7.00 0.00 72.00 20.57 3.84 430.14 126.00 88.20
sdd 10.00 0.00 4.00 13.00 240.00 304.00 64.00 2.07 126.59 33.00 56.10
sda 0.00 1.00 1.00 11.00 16.00 320.00 56.00 0.25 20.42 8.83 10.60
sdc 0.00 0.00 2.00 7.00 32.00 96.00 28.44 0.13 14.89 13.78 12.40
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md3 0.00 0.00 18.00 17.00 288.00 416.00 40.23 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 13.00 0.00 208.00 0.00 32.00 3.66 281.85 21.69 28.20
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-7 0.00 0.00 0.00 5.00 0.00 92.00 36.80 1.84 179.40 189.80 94.90
dm-8 0.00 0.00 0.00 1.00 0.00 4.00 8.00 1.00 655.00 1000.00 100.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.44 1512.00 1000.00 100.00
dm-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 3.00 0.00 16.00 10.67 0.56 6.00 187.67 56.30
dm-18 0.00 0.00 5.00 0.00 80.00 0.00 32.00 0.37 73.40 71.40 35.70
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 0.00 0.00 43.80
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-32 0.00 0.00 0.00 1.00 0.00 4.00 8.00 0.00 1.00 1.00 0.10
dm-33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-35 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-37 0.00 0.00 0.00 7.00 0.00 300.00 85.71 2.00 101.00 142.86 100.00
=================================
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 1.00 1.00 1.00 47.00 48.00 676.00 30.17 1.47 30.62 11.65 55.90
sdd 1.00 0.00 9.00 88.00 176.00 395.00 11.77 0.83 8.59 5.12 49.70
sda 0.00 0.00 0.00 50.00 0.00 541.50 21.66 0.83 16.54 11.22 56.10
sdc 1.00 0.00 6.00 87.00 112.00 533.50 13.88 0.87 9.42 4.37 40.60
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md3 0.00 0.00 21.00 116.00 336.00 991.00 19.37 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 4.00 0.00 64.00 0.00 32.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-8 0.00 0.00 0.00 82.00 0.00 848.00 20.68 1.46 17.93 6.29 51.60
dm-9 0.00 0.00 0.00 3.00 0.00 16.00 10.67 0.03 11.33 11.33 3.40
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-18 0.00 0.00 17.00 0.00 272.00 0.00 32.00 0.49 28.59 10.35 17.60
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-31 0.00 0.00 0.00 5.00 0.00 16.00 6.40 0.13 26.00 26.00 13.00
dm-32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-35 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-37 0.00 0.00 0.00 27.00 0.00 111.00 8.22 1.15 42.41 37.00 99.90
Another example :
Importing an SQL file of 2MB takes 15+ minutes when doing this on a random VM. The I/O for that LV goes too 100% until it's finished. It will only have a couple writes/sec, and just a few KB/s I/O.
Even with these loads, write speeds with dd are around 80MB/s to 160MB/s. But reads are horrible, and most of the time not exceeding 10MB/s.
This is how my RAID setups looks like. This box is running on the "o2" layout, but doesn't shows any difference with the default "n2" layout.
# cat /proc/mdstat
Personalities : [raid10] [raid1]
md3 : active raid10 sdb5[4] sdc5[2] sdd5[3] sda5[0]
3872954368 blocks super 1.2 512K chunks 2 offset-copies [4/4] [UUUU]
md0 : active raid1 sdb1[4] sdc1[2] sdd1[3] sda1[0]
255988 blocks super 1.0 [4/4] [UUUU]
md2 : active raid10 sdb2[1] sdc2[2] sda2[0] sdd2[3]
31454208 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md1 : active raid10 sdb3[4] sda3[0] sdc3[2] sdd3[3]
2096128 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]
This is how the individual disks (sda for example) and LV's are partitioned by each VM. Should be okay for 4K disks.
Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000648b7
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 514047 256000 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 514048 31971327 15728640 fd Linux raid autodetect
/dev/sda3 31971328 34068479 1048576 fd Linux raid autodetect
/dev/sda4 34068480 3907029167 1936480344 5 Extended
/dev/sda5 34070528 3907028991 1936479232 fd Linux raid autodetect
Disk /dev/mapper/raid10storage-kvm396_img: 75.2 GB, 75161927680 bytes
255 heads, 63 sectors/track, 9137 cylinders, total 146800640 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 2097152 bytes
Disk identifier: 0x000a613c
Device Boot Start End Blocks Id System
/dev/mapper/raid10storage-kvm396_imgp1 * 2048 145752063 72875008 83 Linux
/dev/mapper/raid10storage-kvm396_imgp2 145752064 146800639 524288 82 Linux swap / Solaris
I'm using VirtIO on all VM's and the cache is set to "none". Tried different cache settings, but that resulted in even worser performance.
I have a few other servers running for smapper setups (less VM's), but with the same type disks in RAID-1 (2 disks) without any problem.
This really seems to be related to the RAID-10 setups.
I currently have 4 of these RAID10 boxes in use, and all with different CPU and motherboard configurations. But all of these do have the same horrible performance issue.
Does anyone has an idea or theory that could explain why this setups performs so horrible slow?
It's really now workable like this, and it's quite frustrating that I still haven't found the cause after 1 year… 🙁
EDIT – server specs and CPU usage:
All these boxes have different CPU's, but here a few of configurations:
- Xeon E3-1240v2, 32GB DDR3 ECC, SuperMicro X9SCM-F
- Intel Core i7 2600, 32GB DDR3, Intel DQ67OW
- Intel Core i7 3820, 32GB DDR3, Intel DX79TO
And here an output from top, with having one LV @ 100% util and a few others bouncing between 30% and 60%.
Load average are quite low at this moment, but it's quite normal to see them between 10.00 to 30.00 depending on how much LV's have 100% I/O load.
The CPU's are are not really loaded, and there is always at least 30%idle.
top - 02:26:21 up 8 days, 11:34, 2 users, load average: 3.48, 3.99, 3.83
Tasks: 244 total, 2 running, 242 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.3%us, 9.7%sy, 0.0%ni, 62.8%id, 11.4%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 33001640k total, 23715816k used, 9285824k free, 10740k buffers
Swap: 2096124k total, 0k used, 2096124k free, 1159788k cached
The %wa load is at some moments even 50%, but that usually only happens with 5 or more LV's touching the 100%util continuously.
Even with 5 or more LV's running @ 100%, there is no single SATA disk that's really having 90%+ load. They are all having utilization in the 40% to 60% area.
——————————-
EDIT 2, 18-feb-2014;
Today I decided to build another VPS node to do more testing. This system uses the same setup (KVM + SolusVM control panel) as all the other ones. This test setup uses the following hardware:
Intel Core i7 2600
32GB memory
4x WD Black 1.5TB (WD1502FAEX), 7200rpm, 64MB cache, SATA600
2x Samsung 830 Pro 128GB SSD's
So basically a completely different storage layer.
OS: CentOS 6.5 + all recent updates.
Stock kernel kernel, but tried the latest mainline (3.13) kernel too which didn't change the performance.
Created a RAID-10 array (n2 layout, 512K chunks, stock config with bitmaps) on the WD Black drives. Created a RAID-0 array on the SSD drives which will be used by flashcache.
This setup is configured the way that the all LVM I/O goes through the flashcache (as write-through cache). Both the SATA and SSD drives are having the old school 512 sectors, so no extended format. I chose these disks, just to see if there was difference between these and extended format ones which I use most of the time. But I've partitioned them as 4K drives.
I started with some testing on the host node. I created a blank LV volume. While doing some read benchmarks with DD, I had these read results on the host:
- Uncached ~390MB/s read
- SSD Cached ~950MB/s read (second run, when data is cached by flashcache)
Well, these results look pretty good!
Now the same tests, but on a VM with 4GB memory, VirtIO and 'cache=none'. I booted an Ubuntu live DVD, and did the same tests…. this with horrible results:
- Uncached ~90MB/s read
- SSD Cached ~90MB/s read (second run, when data is cached by flashcache)
During the test, the "dm-x" device (LV of the test VM) was loaded all the way up to 95%+, while the individual SATA disks disks didn't go above 35%. Same for the SSD's. No single devices exceeded the 50%util at any moment.
Here is an output of the highest load I've ever mentioned to get on this setup. This was during the uncached test (which explains the writes on the SSDs):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 361.00 2547.00 1444.00 10188.00 8.00 0.62 0.21 0.03 8.90
sdc 992.00 1.00 246.00 7.00 4932.00 14.50 39.10 0.87 3.43 2.16 54.60
sdb 0.00 0.00 303.00 1754.00 1212.00 7016.00 8.00 0.41 0.20 0.03 5.80
sdd 825.00 1.00 173.00 7.00 3992.00 14.50 44.52 0.60 3.34 2.56 46.00
sde 866.00 25.00 229.00 11.00 4404.00 130.00 37.78 0.85 3.50 2.23 53.40
sdf 771.00 25.00 163.00 11.00 3736.00 130.00 44.44 0.65 3.72 2.98 51.80
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md3 0.00 0.00 663.00 4301.00 2652.00 17204.00 8.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md4 0.00 0.00 4266.00 36.00 17064.00 140.50 8.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 4929.00 36.00 19716.00 140.50 8.00 14.56 2.92 0.20 99.60
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 4929.00 36.00 19716.00 140.50 8.00 14.56 2.92 0.20 99.60
(dm-0 is flashcache, dm-2 the guest)
To make sure that the flashcache layer doesn't destroys all performance, I've done the same tests again with flashcache (dm-0) disabled and had pretty much the same horrible 90-100MB/s performance inside the guest.
So for some reason, all performance is gone inside guest. I got these results on a Windows and Linux guest. All using VirtIO and cache set to none. When using IDE the performances drops even further.
Actually a single disk or RAID1 setup with the same flashcache setup performs much better then all MDADM RAID-10 setups @ KVM I used so far.
MDADM RAID-10 performs great on the host, but inside the guest it's horrible.
Right now i'm building a new raid-10 array, but with the offset layout. But I wouldn't expect better performance, as the RAID drive itself isn't fully loaded – it's just the individual "dm-x" LV volumes that are seeing 100% util.
I still can't believe why those RAID-10 + LVM + KVM setups are horrible slow 🙁
Quite frustrating.
Best Answer
You gave us quite no detail on how you use LVM in your setup. If your issue is on reads, their could be two origins.
First of all, check if you're not using LVM snapshots, as they may (will on long term) drastically slow down IO as they grow.
Secondly, your top states that your hypervisor is only using 1G of RAM for IO caches, which isn't really much, although you still have 10G of RAM completely free. This suggests that you tweaked kernel's parameters regarding caches and maybe have worsen performances while trying to things better. As an exemple, here's the memory usage (MB) of one of my hypervisors running 24 VMs with KVM :
As you can see, IO caches use half of the free RAM.