Virtualization Performance – How to Find Bottlenecks?

hyper-vperformanceperformance-monitoringvirtualization

We have recently started moving our C++ build server(s) from real machines into VMs. (MS Hyper-V)

We have some performance issues that I've currently no idea how to address. We have:

  • Test-Box – this is a piece of desktop workstation hardware my co-worker used to set up the VM before we moved it to the actual server hardware
  • Srv-Box – this is the server hardware
  • Test-Box-Real – This is Windows running directly on the Test-Box HW
  • Test-Box-VM – This is Windows in a Hyper-V VM on the Test-Box HW
  • Srv-Box-Real– This is Server2008R2 running on the Srv-Box HW.
  • Srv-Box-VM– This is Windows running in a Hyper-V VM on the Srv-Box HW, i.e. on Srv-Box-Real.

Now, the problem is that we compared Build times between Test-Box-Real and Test-Box-VM and they were basically equal (within about 2%).

Then we moved the VM to the Srv-Box machine and what we saw there is that we have a significant performance degradation between Srv-Box-Real and Srv-Box-VM, that is, where we saw no differences on the Test HW we now do see major differences in performance on the actual Server HW. (Builds about ~~ 50% slower inside the VM.)

I should add that both the Test-Box and the Srv-Box are only running this one single VM and doing nothing else.

I should also note that the "Real" OS is Win2008R2(64bit) and the VM hosted OS is Wind2003R2(32bit).

Hardware specs:

  • Srv-Box:

    • Intel XEON E5640 @ 2.67Ghz (This means 8 cores with hyperthreading on the Real system and "only" 4 cores on the VM, since Hyper-V doesn't allow for hyperthreading, but number of cores doesn't seem to explain the problem here.)
    • 16GB RAM (we have 4GB assigned to the VM)
    • Virtual DELL RAID 1 (2x 450GB HUS156045VLS600 Hitachi 15k SAS drives)

  • Test-Box:

    • Intel XEON E31245 @ 3.3GHz
    • 16GB RAM
    • WD VelociRaptor 600GB 10k RPM SATA

Note again that I'm only concerned with the differences between Srv-Box-Real and Srv-Box-VM (high) vs. the differences seen btw. Test-Box-Real and Test-Box-VM (low).

Why would one machine have parity when comparing VM vs Real performance and the other (server grade HW no less) would have a large disparity? (Both being XEON CPUs …)

Best Answer

Start with the obvious - monitor performance counters, if possible using Ressource MOnitor. How is CPU, IO handling? The standard" bottleneck is pretty much IO, and your 2 drives sound like quite bad IO for anything that has more then 2-3 VM's, especially when something brutal as a build server for C++ hits it (that is HEAVY IO based, compilers in generally are ,but C++ is quite extensive on this area - I use a SSD just for the build).

This is standard Admin - nothing special about a VM at this point. Basically to find the bottleneck you first find it.

IO may by also - dynamic discs. Helps to put in a non dynamic disc ONLY for the build step (work files). There is a TON of IO happening there.

Now, yes, a Raptor has less IOPS than a 15k SAS, but then it is likely NON SHARED.

Does SRV-HW use up all CPU cores? Another item, then...

Regarding your CPU limit, btw., the CURRENT version of Hyper-V supports a LOT more cores than 4 per VM, so an upgrade may be in order... Server 2012 is available for pretty much everyone not buying a box in a shop right now, I got my copy 16th of August from the MS repositories.

But first you do normal operational step by step - find out where the bottleneck is by using ressource monitor and performance

Related Topic