In mode 5, or balance-tlb mode, outgoing traffic uses the MAC address of the slave interface that it's leaving, instead of using the address of the bond interface.
Typically, the bond's MAC is used for all traffic, which can cause a MAC flapping condition between two ports on a given switch - each of your switches will see ingressing traffic with the bond's MAC as the source, both from the direct connection to the device, and from the cross-connect to the other switch.
The transmit load-balancing mode skirts this issue by balancing traffic outbound between interfaces, but by using the interface's MAC address as the source for outbound traffic. If your other nodes in the subnet (particularly the router) don't mind this behavior, then it works just fine - typically there will be no issue, but I can imagine some restrictive router security settings taking offense.
The bond interface will take the MAC address of one of its slave interfaces:
root@test1:~# ifconfig
bond1 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:35
inet addr:192.168.100.25 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe3d:f735/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
eth1 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:35
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
eth2 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:3f
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
eth1's MAC matches the bond interface, it's the "primary", so it's getting the inbound traffic.
And, just to confirm:
root@test1:~# cat /sys/class/net/bond1/bonding/mode
balance-tlb 5
root@test1:~# cat /sys/class/net/bond1/bonding/active_slave
eth1
Ok, so.. is it load balancing? Here's how it looks from another node, sending constant pings:
root@test2:~# tcpdump -e -n -i eth0 proto 1
20:33:08.094078 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 38, length 64
20:33:08.094549 00:0c:29:3d:f7:35 > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 38, length 64
20:33:09.094052 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 39, length 64
20:33:09.094520 00:0c:29:3d:f7:35 > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 39, length 64
20:33:10.094078 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 40, length 64
20:33:10.094540 00:0c:29:3d:f7:35 > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 40, length 64
That all looks normal - eth1 is responding. Then, unprompted, there's a switch - notice that the request's destination MAC and the response's source MAC no longer match.
20:33:11.094084 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 41, length 64
20:33:11.094614 00:0c:29:3d:f7:3f > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 41, length 64
20:33:12.094059 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 42, length 64
20:33:12.094531 00:0c:29:3d:f7:3f > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 42, length 64
20:33:13.094086 00:0c:29:46:4f:c6 > 00:0c:29:3d:f7:35, ethertype IPv4 (0x0800), length 98: 192.168.100.40 > 192.168.100.25: ICMP echo request, id 5810, seq 43, length 64
20:33:13.094581 00:0c:29:3d:f7:3f > 00:0c:29:46:4f:c6, ethertype IPv4 (0x0800), length 98: 192.168.100.25 > 192.168.100.40: ICMP echo reply, id 5810, seq 43, length 64
Watching a constant ping, the switches between source continue arbitrarily based on the bond interface's evaluation of the load - it seems to re-evaluate every 10 seconds.
Failover for inbound traffic in mode 5 is much more basic; when the interface is detected as down, the bond interface's MAC is simply moved over to the live interface. This'll often fire a MAC flapping warning in your switch logs, but that's to be expected; nothing to worry about.
The interface MACs change from this:
eth1 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:35
eth2 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:3f
..to, after taking eth1 down, this:
eth1 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:3f
eth2 Link encap:Ethernet HWaddr 00:0c:29:3d:f7:35
And, all traffic sources from eth2, with a MAC of :35.
So, yeah - assuming that you don't care about load balancing of inbound traffic, the balance-tlb mode seems to do an excellent job of switch-safe load balancing of outbound traffic and failover of inbound traffic.
If your router doesn't care about multiple source MACs sending traffic for a single IP, and doesn't get offended by gratuitous ARP failovers, then you should be good to go!
Absolutely you can. You're looking for 802.1q tagging, which is the underpinning of multiple VLANs on a switch.
Configure your server-facing ports like this...
interface GigabitEthernet0/1
! tell the switch to encapsulate frames with dot1q on this trunk port
switchport trunk encapsulation dot1q
! make this a trunk port unconditionally
switchport mode trunk
! allow VLANs 220, 221, and 223 to be sent on this port
switchport trunk allowed vlan 220,221,223
! send vlan 221 frames untagged
! you could omit this command, but you'll need to move your eth0 config to a subinterface as well
switchport trunk native vlan 221
! this is an end-device, and doesn't need to be treated like a switch for spanning-tree
spanning-tree portfast trunk
! remove the (ignored) access vlan setting, as this is now a trunk
no switchport access vlan
On your Ubuntu server you'll need to configure the subinterfaces to be associated to each appropriate VLAN. This can be done manually like this ...
# create a new subinterface tied to dot1q vlan 221
ip link add link eth0 name eth0.221 type vlan id 221
# change the MTU to allow for the inserted dot1q tag (4 bytes)
ifconfig eth0 mtu 1504
# bring the new interface up
ifconfig eth0.221 up
Now you can associate the correct IP address to new interface eth0.221, along with the appropriate default gateway and subnet mask.
If this doesn't work for you, please include a pastebin link to your switch config as well as the output of "ip -d link show" on your Ubuntu server.
Hope this helps!
Best Answer
I've managed to find the solution. When the Cisco CSR 1000v is booted for the first time, the router interfaces are mapped to the logical vNIC interfaces that were added when the VM was created. The figure below shows the relationship between the vNICs and the Cisco CSR 1000v router interfaces. So, using
show platform software vnic-if interface-mappingI could display the mapping between the logical interface on the router with the vNIC and the vNIC MAC address. Then, in EC2 I created a new network interface, and I attached it to my VM that runs the router. After that, I configured the new interface with the correct destination.Interface IP-Address OK? Method Status Protocol GigabitEthernet1 172.31.xx.xx YES DHCP up up GigabitEthernet2 18.217.xx.xx YES manual up up VirtualPortGroup0 192.168.xx.xx YES NVRAM up up