Bonding/Teaming 2 NIC cards : pro and cons

Okay. After 3 weeks of intense battle I finally got it all figured out.

I opened a case with Broadcom support and after going back and forth for a few days here is the response I received from a Broadcom software developer.

Seeing a (vm_ipv4, host_mac_addr) or (vm_ipv6, host_mac_addr) pair on a remote station or switch is okay for many networks. Basp will handle the muxing of the addresses and make sure the vm's get the correct packets.

However, a few network configurations require the ip address to map to the vm_mac_addr. If you want to use the vm_mac_addr instead of the host_mac_addr, you need to add the HyperVMode registry key listed below.

If it is desired not to mux the vm's mac address, that is to not replace it with a mac address of a nic, you need to set the following registry entry on the host the vm is running on and reboot the system for it to take effect. Setting this registry entry results in a less efficient operation by basp, but is necessary in certain setups, such as when there is a dhcp server that requires the vm's mac address be used in order to assign an ip address. By setting this registry entry, you should also see the actual vm's mac address in the arp table of all the remote stations the vm is connected to.

This registy entry can be added after a team has been created, but before the hyper-v virtual adapter using the team has been created. A value of 1 will configures basp to use a vm's mac address instead of the mac address of a nic in a slb team when load balancing flows. A value of 0, or not present, configures basp to mux/demux the vm's mac address with a mac address of one of the nics in the team.

Set the "1" in the registry path to the correct team number that this mode is to be applied to and reboot the system. Note that this registry entry will be deleted when the team is deleted, and must be recreated each time the team is created. Future versions of BACS will have a check-box that will set this registry entry and prevent the need for adding it when a team is created and having to reboot the system. This registry entry is only available on >=basp6-1.5.1.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ Blfp\Parameters\1] "HyperVMode"=dword:00000001

PS. Besides this registry entry being only available on >=basp6-1.5.1, it is only applicable to slb team types, and does not apply to lacp or gec teams.

So with that said I gave it a try. First, I applied created the registry key. Then, I changed the Virtual Switch to Private Virtual Machine Network mode. I gave the server a quick reboot. Finally, when the server came back online I configured the Virtual Switch back to External Mode and choose the BASP Virtual Adapter. I tested Live Migrations and everything worked perfectly and the ARP tables in the Nexus showed the VM_IP and the VM-MAC. YEY!!!!

It is defenitely interesting question and here is how I did a trick. My configuration is about the same as yours: Proxmox booting via iSCSI bridged interface. My grub boot line looks like this:

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 ISCSI_INITIATOR=iqn.2007-08.com.example.client:client ISCSI_TARGET_NAME=iqn.2005-10.org.freenas.ctl:proxmox ISCSI_TARGET_IP=192.168.11.3 ISCSI_TARGET_PORT=3260 root=UUID=04709453-9d82-47d6-a898-81ea6408f88e ip=192.168.11.1:::255.255.255.0:client:vmbr2:off"

To boot with this grub configuration I need to start the bridge before grub mounts root. I make it by assigning a script in /etc/udev/rules.d/70-persistent-net.rules like this:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:48:94:61:ec", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="ens6f0", RUN+="/bin/brctl-iscsi ens6f0 vmbr2"

this /bin/brctl-iscsi script looks like this:

#!/bin/sh

ifconfig $1 mtu 9000 up
brctl addbr $2
brctl addif $2 $1
ip link set dev $2 type bridge forward_delay 0 stp_state 0

I also need a initramfs hook to put this script in initrd:

root@proxmox-2:~# cat /usr/share/initramfs-tools/hooks/brctl_iscsi
#!/bin/sh -e

PREREQS=""

prereqs() { echo "$PREREQS"; }

case "$1" in
    prereqs)
    prereqs
    exit 0
    ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /bin/brctl-iscsi    /bin

And finally in /etc/network/interfaces I have this for iscsi interface and its bridge:

no-auto-down ens6f0
no-auto-down vmbr2

iface ens6f0 inet manual

iface vmbr2 inet manual
    address  192.168.11.2
    netmask  255.255.255.0
    bridge-ports ens6f0
    bridge-stp off
    bridge-fd 0
    down ifconfig vmbr2 down; brctl delbr vmbr2
#iSCSI network

That's it Regards, Dmitry