I have Two HPBL685c G6 blade servers running Ubuntu 15.04
- Each server has 4X10GB NIC
- 2x10GB NIC are connected to a single VirtualConnect 10/10G Ethernet module
- The other 2x10GB NIC are connected to a second VirtualConnect 10/10G Ethernet module
- Virtual connect modules are configured for Horizontal Stacking, and are located in interconnect bays 1 & 2
- The NICs being referenced are all embedded Flex-10 adapters
When I configure each of the 4 10GB NIC separately, I can test using iperf and I receive ~10Gbit/sec bandwidth between servers, for each NIC. This works as expected.
- Server 1: http://d.pr/n/15dA5
- Server 2: http://d.pr/n/10pqJ
- iperf results: http://d.pr/i/pscU and http://d.pr/i/zh2E
Now, I am trying to bond all of the 10GB NIC on each server, using bond mode "balance-rr". These results vary, but land somewhere between 2.5Gbits/sec and 5Gbits/sec
- Server 1: http://d.pr/n/1aTei
- Server 2: http://d.pr/n/12Mqy
- iperf results: http://d.pr/i/1cvh0 and http://d.pr/i/1eOgU
I am using the same configuration to bond 2X1GB NIC on these same servers, 2X1GB NIC bonded results in ~2Gbit/sec bandwidth when testing with iperf. These 2 NICs are not connected to a Virtual Connect Domain, and instead are each connected to a different Cisco Catalyst Blade Switch 3120
- Server 1: http://d.pr/n/1kk4f
- Server 2: http://d.pr/n/fbVJ
- iperf results: http://d.pr/i/10N4q and http://d.pr/i/1a0z3
So, my question is: Why does bonding 4X10GB NIC using balance-rr, result in less performance than using a single NIC. I would have expected ~40Gbit/sec bandwidth minus TCP/Bonding overhead, which would align with my results when bonding 2X1GB and getting ~2GB when testing.
I have tried this with different bonding modes, and the others result in about ~10Gbit/sec bandwidth when bonded. Still not ideal, but better than the balance-rr results.
Best Answer
It appears that the Virtual Connect Modules do not support bond mode 0 (balance-rr) in linux deployments.
From HP Support: http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c02957870
Information Unsupported bonding modes in an HP Virtual Connect environment can produce packet loss and/or performance issues.
Details HP Virtual Connect supports bonding modes 1, 5, or 6. VC does not support modes 0 (round robin) or 7 (switch assisted load balancing).
Mode 1: Active/backup. Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.
Mode 5: Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.
Mode 6: Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation.