I had a similar problem trying to raise the speed of a drbd synchronization over two gigabit links some time ago. In the end I managed to get about 150MB/sec synch speed. These were the settings that I applied on both nodes:
ifconfig bond0 mtu 9000
ifconfig bond0 txqueuelen 10000
echo 3000 > /proc/sys/net/core/netdev_max_backlog
You could also try to enable interrupt coalescence if you don't already have for your network cards (with ethtool --coalesce)
LACP itself doesn't provide the ability to bond across multiple switches; it bonds across multiple ports on a single ethernet switch, and depending on the vendor there might even be restrictions on which ports on a switch can be bonded together.
Some vendors have proprietary protocols (typically called MLAG) that allow for bonded ethernet channels across different ethernet switches. As an example Cisco Nexus vPC (or generically MLAG) works with switches, or bonding a single LACP port channel on a server across two connected switches.
Does the use of bonded ethernet channels across multiple switches (that we are advised that we can use) from the server, provide both improved throughput (unquestionably), and improved redundancy (uncertain). Could/would network events such as switch failure, port migration, patching, recovery, etc, cause the channel for both server network interfaces to be unavailable?
LACP should provide protection against a single physical port or cable failure within the LACP channel.
LACP cannot protect against human factors, such as accidentally shutting down the LACP-interface, removing the vlan, or running a TDR on a port-channel member link. LACP also cannot protect against over-provisioning bandwidth through a single member link on that LACP channel, spanning-tree events, broadcast storms, excessive unknown unicast flooding, etc...
If you are concerned about recovery time, be sure to use short LACP protocol timeouts on your interfaces.
Best Answer
The document on Linux bonding is worth the read, it goes into what sort of support you need from switches for various bonding methods. In this case:
So you will need to group the ports on your switch (often just creating a LAG). Gets a bit more involved sometimes though if you want to plug the same bond into multiple switches.
If you don't want the switches involved you probably want
balance-albwhich includes both transmit and receive balancing: