Best way to benchmark linux virtualization solutions

benchmarkkvm-virtualizationopenvzvirtualizationxen

I'd like to run some benchmarks on linux virtualization solutions (namely Xen, KVM and OpenVZ) and I'm currently thinking about how to set up the test environment. My current approach would be the following:

  • Set up a bare server system
  • Install all 3 solutions (kernels) on the same system and set them up (+ VMs)
  • Boot a specific kernel (disable eventual services from other solutions) and run benchmarks

Would this approach be OK or should I completely separate the 3 systems from each other (have 3 completely independent installations each one with only one solution installed)?

Additional question: any hints what kind of benchmarks I should run? I thought about some compiling (e.g. linux kernel), ab (Apache Benchmark) and OSDB (Open Source Database Benchmark). The comparison should mainly focus on overall performance, I/O performance and response time of individual VMs.

Thanks in advance,
Mathias

Best Answer

Testing should be done as close to how you'd run it in production. So install each system separately and do the testing. This may require flattening your testing box several times, but it will give the cleanest results. Then focus on creating your test cases. Some off the top of my head:

  • Multiple VM test Run several VM's on the same host.
  • Busy VM resource contention Have one VM run something highly CPU demanding, run other tests in other VMs to see how they're affected by it.
    • Do the same, but with a high disk I/O VM, and see how other VMs are affected.
  • Multi-processor efficiency If you have the hardware for it, run VM's with more than one virtual CPU and judge efficient use of it (say, kernel compile with multiple jobs).

In essence, brain-storm the edge cases and build tests around those cases. What happens when a VM gets piggy with resources? How well does the hypervisor handle over-subscription of resources? How does one VM locking up hard affect other VMs on the system?

Related Topic