Azure – Network Interface Physical MAC Address

This is something that may be unique to the HP blade chassis setup. Since you mention that this is a blade server, do you know if the blade chassis has HP Virtual Connect modules installed? These are often used instead of passthrough ethernet uplinks to help consolidate and virtualize the blade networking.

If you do have Virtual Connect modules, check the server profile in the Virtual Connect Manager. There are options to manage the server MAC addresses using Virtual Connect. See page 64 of the Virtual Connect Installation Guide for an explanation of the feature:

Each server blade Ethernet NIC ships with a factory default MAC address. The MAC address is a 48-bit number that uniquely identifies the Ethernet interface to other devices on the network. While the hardware ships with default MAC addresses, Virtual Connect can assign MAC addresses that override the factory default MAC addresses while the server remains in that Virtual Connect enclosure.

The other answers posted certainly apply to multiple NICS in physical rack/tower servers, but the HP blade solution is a very special case.

The book is correct, however there are pieces it left out.

• MAC addresses are not as fixed as you would think, most higher end NICs have the ability to change the MAC address to something specific. Either in the NIC's BIOS or in the driver itself.
• There are specific ranges of MAC address set aside for 'virtual' systems (see What range of MAC addresses can I safely use for my virtual machines?)
• Clustering software may use MAC addresses from these set-aside ranges to present the cluster IP services.
• The linux network stack has the ability to create virtual NICs with specific MAC addresses.

The procedure followed by clustering software to create a service based on a virtual MAC address is pretty straight forward. When the service comes up, it offers a Gratuitous Arp packet saying that the specific IP address can be found on the virtual MAC address. When failover happens, the 'down' node removes its local IP/MAC binding and the new node starts listening to that virtual MAC address and IP combination. No muss no fuss.

The other method used by clustering software is to not bother with virtual MACs at all and rely on Gratuitous ARP the whole way. The startup/failover sequence for such a system would look like:

1. cluster software binds IP to Node A.
2. Node A G-ARPs "192.168.244.60 is on 02-00-ab-cd-ef-01"
3. All devices on the subnet update their ARP tables.
4. Time passes.
5. Node A crashes.
6. Cluster software binds IP to Node B.
7. Node B G-ARPs "192.168.244.60 is on 02-00-ab-cd-41-ba"
8. All devices on the subnet update their ARP tables.

In my experience the second method, pure G-ARP, is the one used by most linux clustering these days. However, both methods are valid and have been used. The benefit of the G-ARP method is that you don't have to muck about assigning virtual MAC addresses. The benefit for the pure virtual-MAC method is that it doesn't rely on G-ARP working on a given subnet.