Xen or KVM? Please help me decide and implement the one which is better

Testing should be done as close to how you'd run it in production. So install each system separately and do the testing. This may require flattening your testing box several times, but it will give the cleanest results. Then focus on creating your test cases. Some off the top of my head:

• Multiple VM test Run several VM's on the same host.
• Busy VM resource contention Have one VM run something highly CPU demanding, run other tests in other VMs to see how they're affected by it.
  • Do the same, but with a high disk I/O VM, and see how other VMs are affected.
• Multi-processor efficiency If you have the hardware for it, run VM's with more than one virtual CPU and judge efficient use of it (say, kernel compile with multiple jobs).

In essence, brain-storm the edge cases and build tests around those cases. What happens when a VM gets piggy with resources? How well does the hypervisor handle over-subscription of resources? How does one VM locking up hard affect other VMs on the system?

We use Xen currently, but I think that in 2011 we will be migrating to KVM. There are some reasons why:

• KVM development is more integrated with the Linux kernel than Xen's development.
• KVM vms run as processes to the linux kernel. That has many implications, like scheduling, memory management and etc. That also permits KVM to do overcommit of memory (it's simply swapping the vm/process memory) and all that is using tested and proven code from the kernel. Xen uses it's own code to do all that, and while it's not bad, it's not as tested and proven as the linux kernel.
• Full virt seems to work better on KVM.

The main point against KVM is performance, but last reports using VirtIO driver on Linux and Windows VMs seems to make that point less and less proehminent.

As for managing the machines, I use ganeti. Ganeti is a cluster virtual server management system where you add your nodes and can do all the operations with the VMs on those nodes like creating, starting, rebooting, migrating and etc. It also supports the creation of DRBD instances that have mirrored disk images on two nodes providing automatic failover/migration if one of the nodes fail. It supports KVM or Xen, but not mixed clusters. Ganeti is text-based but there's a web interface project that is doing pretty well. Using ganeti + debootstrap we can deploy/clone various types of vms very fast, do LVM snapshots from disks for tests and so on, so I think it will get you covered on that.

Remember only that no matter what one you choose, you should never virtualise high IO machines like file servers or DBs and expect they will perform the same. In some cases, high IO VMs can even degrade all other VMs on a host. Not everything can be safely virtualised.

EDIT: Since you mentioned testing and development, read this article. I am thinking in assembling something very close to that.