OK, after a long time trawling the Internet and trying to read through man pages, I mostly figured out what's going on:
[H]ow in the heck did it know to send
the server's real hostname if the
configuration isn't set up to send it
in the first place?
Apparently, send host-name "<hostname>", and more specifically, <hostname>, has special significance on Ubuntu. It tells dhclient to send your machine's current hostname to the DHCP server. Note that this is case-sensitive (i.e. if you type <HOSTNAME>, dhclient will send the literal text <HOSTNAME> to the DHCP server). As reported here and here, this works on Ubuntu Feisty and above, being part of a patch to (ironically) fix a problem with dhclient not sending the machine's hostname by default.
[W]hy did the first DHCP lease (for the
old network) not include option
host-name, but the second DHCP lease
(on the new network) did include it,
if I haven't touched any of the
configuration files?
This just seems to be because not all DHCP servers will echo the hostname back to the client. Yesterday we still using a Linux-based ClarkConnect router as our DHCP server, which echoed the hostname back to each client. Today we disabled DHCP on that server, and switched to using the built-in DHCP server on our primary domain controller. Our DHCP leases from the PDC don't include option host-name either, but it doesn't seem to affect anything. The machines get updated in DNS, and they can all find each other by hostname.
When I run hostname, the server
returns fozzie.our.domain, but
dhclient.eth0.leases says the hostname
option was set to fozzie (no domain).
How did it know to strip off the
domain?
This is me not fully understanding the intricacies of DHCP. Our DHCP server was set to configure all of our clients with our domain. I'm guessing the DHCP server was smart enough to remove the domain name portion from the hostname sent by the client.
Anyone have any corrections or clarifications to add to this answer? It still seems a bit "hand-wavy", so I won't accept it as-is.
So basically you want your client to send their hostname to your DNS server ?
This works great with Windows clients, but for Linux machines, I would go the other way around : set your clients to ask the DHCP server for an IP and a hostname. Using the "request host-name" option, dhclient asks the DHCP server to send a hostname along with the IP address, then sets the hostname of the VM.
Of course you'll need to prepare, in your DNS, a hostname for each IP address in the DHCP range.
This way you don't have the DNS cache / propagation delay problem, and you may have a central place with all the IP and hostnames - your DNS server.
I don't know wether your appliance's DHCP server supports sending hostnames - most does, but I think that's the easiest way for Linux clients.
Note: If you configured your DHCP server so that the same machine always get the same IP address (e.g. using mac address) then it will, obviously, always get the same hostname too.
On some DHCP server, you can use directly hostname instead of IP address in the config files.
Best Answer
Why not have two IP addresses on a single client interface and send in two DHCP-with-hostname requests, use a reservation if you want to be consistent. Obviously it's a waste of an IP but it should work just fine.