Xen Cloud Platform

I'm not an expert on Xen migration, and I'm using the open-source Xen server. In my experience, the Xen server is very efficient with migration as long as your storage layer is fast -- in our experience, disk images as files on ocfs2 volume or (god forbid) NFS mount were much slower than block devices on a SAN with a shared locking volume on a NFS mount. We haven't had problems with disk corruption, but do tend to snapshot things (both LVM2 and VM state) before we start the migration on a very active system just to be sure.

According to "Running Xen: A Hands On Guide to the Art of Virtualization" by Matthews, Dow, et al., Prentice Hall 2008, page 484,

The implementation of Xen's live migration involves the novel use of an iterative multipass algorithm that transfers the virtual machine guest memory in successive steps. After the source VM and the destination VM first negotiate to ensure resources are sufficient on the receiving machine, an initial pass over the guest's memory is performed with each page being transferred to the destination. On each successive iteration, only the guest memory that has been dirtied in the interim is sent. This process is executed until either the remaining number of dirty pages is sufficiently small enough(sic) that the remaining pages can be transmitted quickly or the number of dirty pages remaining to transfer in each pass is not decreasing. At that point, the system is actually quiesced and the final state sent to the new host and transfer of control to the new physical machine completed.

It looks like that's similar to the list of steps you described above, with the addition of the iterations. Note that the machine may be doing I/O at two places with the current state of live migration.

Unlike VMWare and HyperV, the nice thing about XenServer is that there's a ton of people that have been running it and trying their hardest to break it ten ways from Sunday in very serious production environments. Live migration is new to us and we don't do it in a production environment yet because we have redundancy concerns (non-trivial to scale to n machines at this point due to having our shared data partitions on ocfs2 volumes), but in our test environments we've been having fun bouncing machines all over the place.

"Best" is always very subjective, as is inexpensive, but for free the best options are Citrix XenServer and Oracle VM.

They're both based on Xen, both have management GUIs, various tools, etc. In terms of functionality of these commercial but free products, I think only Oracle VM supports both live migration and high-availability in it's free version.

For your other questions, the performance difference between virtualised and "bare-metal" performance is typically around 5%-10%, not a significant amount really compared to the management benefits. The hit is often in the area of disk I/O rather than CPU usage.

You're right that often these implementations have a single non-virtual management server, but generally there's no reason why they can't be virtual themselves, it's just that tracking it down in the event of a failure can be tricky - you end up asking the question "which box has it ended up on when it failed over?". Assuming you setup the correct failover rules, you'll be able to work it out without a problem, and with only 5 physical hosts it's not a huge issue to track it down anyway.