Managing XenServer and its Virtual Machines Remotely

"Best" is always very subjective, as is inexpensive, but for free the best options are Citrix XenServer and Oracle VM.

They're both based on Xen, both have management GUIs, various tools, etc. In terms of functionality of these commercial but free products, I think only Oracle VM supports both live migration and high-availability in it's free version.

For your other questions, the performance difference between virtualised and "bare-metal" performance is typically around 5%-10%, not a significant amount really compared to the management benefits. The hit is often in the area of disk I/O rather than CPU usage.

You're right that often these implementations have a single non-virtual management server, but generally there's no reason why they can't be virtual themselves, it's just that tracking it down in the event of a failure can be tricky - you end up asking the question "which box has it ended up on when it failed over?". Assuming you setup the correct failover rules, you'll be able to work it out without a problem, and with only 5 physical hosts it's not a huge issue to track it down anyway.

If the VM images are on a separate VG, all you need to do is avoid formatting it when reinstalling - go for manual partitioning and make sure the "virtualization" VG is not touched

What you need to preserve is the VM configs, because besides the image there's all the setup data - usually in /etc/libvirt/qemu/guestname.xml