What you're looking for is commonly called a "transmit hash policy" or "transmit hash algorithm". It controls the selection of a port from a group of aggregate ports with which to transmit a frame.
Getting my hands on the 802.3ad standard has proven difficult because I'm not willing to spend money on it. Having said that, I've been able to glean some information from a semi-official source that sheds some light on what you're looking for. Per this presentation from the 2007 Ottawa, ON, CA IEEE High Speed Study Group meeting the 802.3ad standard does not mandate particular algorithms for the "frame distributor":
This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 43.2.3, the algorithm shall not cause a) Mis-ordering of frames that are part of any given conversation, or b) Duplication of frames. The above requirement to maintain frame ordering is met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to re-order frames.
So, whatever algorithm a switch / NIC driver uses to distribute transmitted frames must adhere to the requirements as stated in that presentation (which, presumably, was quoting from the standard). There is no particular algorithm specified, only a compliant behavior defined.
Even though there's no algorithm specified, we can look at a particular implementation to get a feel for how such an algorithm might work. The Linux kernel "bonding" driver, for example, has an 802.3ad-compliant transmit hash policy that applies the function (see bonding.txt in the Documentation\networking directory of the kernel source):
Destination Port = ((<source IP> XOR <dest IP>) AND 0xFFFF)
XOR (<source MAC> XOR <destination MAC>)) MOD <ports in aggregate group>
This causes both the source and destination IP addresses, as well as the source and destination MAC addresses, to influence the port selection.
The destination IP address used in this type of hashing would be the address that's present in the frame. Take a second to think about that. The router's IP address, in an Ethernet frame header away from your server to the Internet, isn't encapsulated anywhere in such a frame. The router's MAC address is present in the header of such a frame, but the router's IP address isn't. The destination IP address encapsulated in the frame's payload will be the address of the Internet client making the request to your server.
A transmit hash policy that takes into account both source and destination IP addresses, assuming you have a widely varied pool of clients, should do pretty well for you. In general, more widely varied source and/or destination IP addresses in the traffic flowing across such an aggregated infrastructure will result in more efficient aggregation when a layer 3-based transmit hash policy is used.
Your diagrams show requests coming directly to the servers from the Internet, but it's worth pointing out what a proxy might do to the situation. If you're proxying client requests to your servers then, as chris speaks about in his answer then you may cause bottlenecks. If that proxy is making the request from its own source IP address, instead of from the Internet client's IP address, you'll have fewer possible "flows" in a strictly layer 3-based transmit hash policy.
A transmit hash policy could also take layer 4 information (TCP / UDP port numbers) into account, too, so long as it kept with the requirements in the 802.3ad standard. Such an algorithm is in the Linux kernel, as you reference in your question. Beware that the the documentation for that algorithm warns that, due to fragmentation, traffic may not necessarily flow along the same path and, as such, the algorithm isn't strictly 802.3ad-compliant.
I have finally managed to craft a working solution.
- Solution uses ebtables and IP-MAC pairs.
- Only needed table is the default 'filter' table.
- There is no need to add any rules or policy to INPUT chain, since INPUT chain is NOT related to running virtual machines. Explanation of meaning of INPUT, OUTPUT and FORWARD chains in filter table is in ebtables manpage.
- Since ebtables works on ethernet level and the IP-MAC pairing is aplicable only for IP packets, there is need to emphesize that in the rules in order not to block ARP frames and other vital traffic.
So, in the begining, there are no rules whatsoever and all policies are set up to ACCEPT. There are no user defined chains. Filter table looks like this:
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
A new chain is addded. This chain contains all the allowed IP-MAC pairs. It is called VMS.
# ebtables -N VMS
Now, the important part. For every frame containing IP packet (or its parts) which is going thru bridge from port vnet[0-9]+, apply chain policy and rules of chain VMS. In other words, for every IP packet comming from any virtual machine, apply VMS chain.
# ebtables -A FORWARD -p ip -i vnet+ -j VMS
The default policy of chain VMS has to be DROP. This way, every IP packet comming from any virtual machine is droped by default. Later, allowed IP-MAC pairs exceptions are added. The default policy DROP causes, that all traffic from any virtual machine with unknown IP-MAC pair is droped at once, making IP spoofing impossible.
# ebtables -P VMS DROP
The table filter looks now this way. Also, this way it looks when there are no virtual machines running (allowed).
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 1, policy: ACCEPT
-p IPv4 -i vnet+ -j VMS
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge chain: VMS, entries: 0, policy: DROP
Suppose, there are two running machines. If we try to ping to/from them, traffic is dropped and destination is unreachable. This is desired result, as of this traffic has not been allowed yet. Only one command is enough to allow every one virtual machine traffic.
# ebtables -A VMS -p ip --ip-src 192.168.11.125 -s 54:52:00:cc:35:fa -j ACCEPT
# ebtables -A VMS -p ip --ip-src 192.168.11.122 -s 54:52:00:98:d7:b6 -j ACCEPT
Now, traffic from allowed virtual machines is flowing alright and IP spoofing is prevented.
This solution may be unperfect and if you have any comments or improvements, I will gladly hear them.
Best Answer
You would need an intelligent switch to do this on the network. I don't know if there is a VM management tool that can prevent users from adding another or changing IP's on their guest.
In Cisco-land you're looking for port-security to keep someone on a different port from spoofing the mac, and IP source guard (IPSG) to inspect the mac/IP combination and/or dynamic arp inspection (DAI) to prevent ARP spoofing. IPSG and DAI depend on DHCP snooping or a user-configured table, so it can add quite the overhead to your operation.
Other vendors(Juniper/Extreme/Force10/etc...) can do the same security features but the names might differ from what I mentioned. All vendors will have their own hardware/software/licensing requirements that you will need to work out with your vendor/VAR.
Also, there are complications to configuring this type of security and a misconfiguration can be really hard to troubleshoot, and depending on your network it might not be possible to provide this level of security.