In addition to the info on Why do people tell me not to use VLANs for security? here are some more specific & general bits to consider:
General Thoughts on Security
The most secure system is one where each subnet's hosts are connected to a switch having exactly the number of ports that will be used by connected devices. In such a configuration you can't plug random machines into your secure networks, since doing so would require unplugging something (and theoretically your monitoring system would notice that).
VLANs give you something similar in terms of security, breaking your switch up into smaller virtual switches (virtual LANs: VLANs) that are isolated from each other logically, and with proper configuration can appear to all the systems connected to them as if they were physically isolated.
General Thoughts on relatively-secure VLAN Setups
My practice for VLAN-capable switches is that all traffic must be assigned to a VLAN, with the following basic configuration:
Assign all unused ports to an "unused" VLAN.
All ports connecting to a specific computer should be assigned natively to the VLAN that computer should be in. These ports should be in one and only one VLAN (barring certain exceptions that we'll ignore for now).
On these ports all incoming packets (to the switch) are tagged with the native VLAN, and outgoing packets (from the switch) will (a) only originate from the assigned vlan, and (b) be untagged and appear just like any regular ethernet packet.
The only ports that should be "VLAN trunks" (ports in more than one VLAN) are trunk ports -- those carrying traffic between switches, or connecting to a firewall that will split up the VLAN traffic on its own.
On the trunk ports the vlan tags coming in to the switch will be respected, and vlan tags will not be stripped from packets leaving the switch.
The configuration described above means that the only place you can easily inject "VLAN hopping" traffic is on a trunk port (barring a software problem in your switches' VLAN implementation), and much like in the "most secure" scenario this means unplugging something important and causing a monitoring alarm. Similarly if you unplug a host to connect to the VLAN it lives in your monitoring system should notice that host's mysterious disappearance and alert you.
In both of these cases we're talking about an attack involving physical access to the servers -- While it may not be completely impossible to break VLAN isolation it is at a minimum very difficult in an environment set up as described above.
Specific Thoughts on VMWare and VLAN Security
VMWare Virtual Switches can be assigned to a VLAN -- When these virtual switches are connected to a physical interface on the VMWare host any traffic emitted will have the appropriate VLAN Tag.
Your VMWare machine's physical interface would need to be connected to a VLAN trunk port (carrying the VLANs that it will need access to).
In cases like this it is doubly important to pay attention to VMWare Best Practices for separating the Management NIC from the Virtual Machine NIC: Your Management NIC should be connected to a native port in an appropriate VLAN, and your Virtual Machine NIC should connect to a trunk that has the VLANs the virtual machines need (which ideally shouldn't carry the VMWare Management VLAN).
In practice enforcing that separation, in concert with the items I mentioned and what I'm sure others will come up with, will yield a reasonably secure environment.
You could generate a UUID from the host when the VM is created, and pass it in via the kernel command-line. When you transfer the machine from one hardware node to the next, the UUID would travel with it. However, when you create a new machine, a new UUID would be generated for it.
As mentioned in the other answer, this won't work if you duplicate the VM (including its configuration), but in that event it wouldn't be hard to run uuidgen from the command line and replace the UUID in the configuration file that is used to create the domain.
As a matter of practice, it's probably best not to duplicate VMs in that manner anyway; you should create a new VM and have a script run that configures the VM in whatever manner best fits your environment.
Best Answer
You're thinking about this the wrong way. Instead of trying to derive the KVM hostname from the guest (impossible by design), try to figure out how the KVM host might inform the guest of its name.
Data can be easily shared with a guest from a KVM host by using the Filesystem Passthrough feature. There's even a GUI option for it in Virtual Machine Manager (See Add Hardware -> Filesystem).
Just create a directory somewhere on the KVM host with whatever stuff in it you want to share with the guest. This can be a text file containing the hostname, some setup or maintenance script, or whatever you want really. Mark it read-only for simplicity and security. Then configure Filesystem Passthrough using this path and mount it somewhere convenient in each guest.
Obviously this approach requires slight modifications to each KVM host and guest, but with a few hundred hosts out there, I assume you have some sort of configuration management in place (like Puppet, Ansible, etc.), thus greatly simplifying the matter.
Sorry this answer is super late, but given that the question has 5,000+ views I figured I should post it anyway.