Which is the recommended filesystem for VMware Server / ESXi

No problem, this can be done easily using 'raw device mapping' or by using VMFS on the disk - plus the P400 has a native driver in v4 too. You're onto a winner :)

As free stand-alone products both do a pretty similar job; they work, are fast and quite feature-rich.

What you need to ask yourself is how you wish to manage and automate the work involved in the lifespan of the client VMs, also how many VMs you wish to serve per host server and what your client SLAs will be.

The reason you need to ask yourself these questions is that their answers will define how serious you are about this business and therefore which solution to go for.

For instance neither free product comes with serious management tools or support, you pay for that with either option. In my own experience the management, and especially the automation, tools for VMWare outstrip Xen's so that may be of interest to you if you expect to orchestrate VM creation/deletion. VMWare will also allow you to host more VMs per host server too due to it's use of shared memory. Also VMWare has a more mature set of in-house and third-party tools available to help you automatically measure client SLAs, although as Xen matures it is gaining ground in this area too.

Either way you need to decide how serious you are about this business and invest wisely, I'd advise you invest in some sensible shared storage of some time to allow for live VM migration and/or automatic VM restarts in the event of host server failure.

Getting to your actual question; I'd go with VMWare but mostly because I know it pretty well, although I will caveat that by saying I wouldn't be particularly bothered if I were forced to use Xen (oh and have you considered KVM by the way?)

Good luck.