Better MS HyperV or VMWare ESX

VMWare ESX or ESXi CAN run inside a virtual machine, provided certain prerequisites are satisfied.
This kind of setup is of course completely useless (and totally unsupported) in a production environment, but can be very useful for two purposes:

• Testing or studying ESX or ESXi if you don't have a physical server available.
• Testing or studying the whole Virtual Infrastructure if you don't have at least two servers and a SAN.

Prerequisites:

• You need some physical resources. In order to run ESX or ESXi in VM, the VM needs at least 1.5 GB of memory, two VCPUs and enough disk space for the server itself and for the VMs you will run inside it.
• You absolutely need a physical CPU with native virtualization support (Intel VT or AMD-V).
• You need to run VMWare Workstation 6.5, VMWare Server 2 or VMWare Fusion 5 on the physical host. Previous versions can't succesfully run ESX or ESXi in a VM.
• A 64-bit OS on the physical host is useful but not required.

Setup:

• Enable native virtualization support for your CPU in the motherboard BIOS (it's often not enabled by default).
• Install your preferred virtualization software. I've tested everything succesfully using VMWare Workstation 6.5.2 on a Windows XP x64 host, but it should work with VMWare Server 2.0 and/or Linux hosts, too.
• Create a custom VM using those setting:
  • Hardware compatibility level: latest
  • Guest operating system: other 64-bit
  • Virtual CPUs: at least 2
  • Memory: at least 1.5 GB
  • Networking: Host-only or NAT
  • SCSI adapter: LSI Logic
  • Virtual disk type: SCSI
  • Virtual disks: as you wish; I suggest using at least two virtual disks, a 10-GB one for installing the system and another one where to create a datastore. The space should be pre-allocated.
  • Remove floppy, sound card, USB controller, etc. Leave only networking and storage.
  • CPU Execution mode: Intel VT-x or AMD-V (very important).

• Manually edit the VMX file of the virtual machine you created, setting the following parameters:

  guestOS = "vmkernel"
monitor_control.vt32 = "TRUE"
monitor_control.restrict_backdoor = "TRUE"

• Start the VM and install ESX or ESXi from the installation ISO image.

• Configure the networking to allow the ESX or ESXi virtual server to talk with the host.

Usage:

• Use your web browser to connect to you virtual server's IP address and download the VI Client.
• Install the VI Client on the host.
• Connect to the virtual ESX/ESXi server.
• Create a VM as you wish.
• Power up the VM.

• If everything is done correctly, the VM will start. If it complains about not being able to power on a VM inside a VM, then there is an error with the `monitor_control.restrict_backdoor' parameter (or you're using an old version of VMWare Workstation/Server).

• Enjoy :-)

It is hard to tell from your description, but I think what you are seeing is that the MAC address of the service console is different than the MAC address used by the server during the installation. ESX can also have assigned IP addresses for VMotion/VKernel services.

It sounds to me like you should adjust your DHCP configuration post-install to give the proper IP address to the new MAC address because the physical interfaces in ESX could be used for multiple purposes (VM networking, iSCSI, service console, VMotion, etc)

If adjusting the DHCP configuration isn't what you are looking for, you can change the MAC address of a virtual interface by editing the /etc/sysconfig/network-scripts/ifcfg-vswif0 file at the service console and restarting networking (at least in 3.5)

NOTE: The vsiwf# may be different in your environment so be sure you are editing the correct file for your configuration.