I have a FreeNAS Server with four network interfaces. The iSCSI traffic goes through two of those interfaces, and each interface has one IP address in a different subnet. As example:
igb0: 192.168.10.1/24
igb1: 192.168.11.1/24
There are three XenServer hosts with only one interface dedicated to iSCSI traffic. So in the schematic are two interfaces on the storage and three as the total on the hosts.
My plan was to achieve up to 2Gbit connection with the hosts, limited as 1Gbit per host.
The problem starts with the different subnetting. I don't know how to put two different IP's addresses on the same network interface on the XenServer hosts. The XenCenter simply don't let me to do this. Another idea was to isolate this kind of traffic with different VLANs. It's OK, but this does not appears to work, either.
EDIT:
Unfortunately LACP does not work as expected, there are more info on FreeNAS docs:
"LACP and other forms of link aggregation generally do not work well with virtualization solutions. In a virtualized environment, consider the use of iSCSI MPIO through the creation of an iSCSI Portal. This allows an iSCSI initiator to recognize multiple links to a target, utilizing them for increased bandwidth or redundancy. This how-to contains instructions for configuring MPIO on ESXi."
That's why I'm trying to setup MPIO even with VLANs and hacks to achieve the 2Gbit/s for the storage.
Best Answer
If each host only has one interface for iSCSI, then you won't be able to use MPIO with the setup you've described here. However, you should be able to configure the FreeNAS system to use Link Aggregation (LACP), so that you can service two hosts simultaneously each at 1Gb (for a total of 2Gb from the FreeNAS). Instead of MPIO, look into LACP (or, get a second NIC for each host).
EDIT: The reason that LACP is generally not recommended for virtualization is because it doesn't do what people expect. They usually expect that by putting two NICs on a host and two NICs on the storage, they can double the bandwidth to the storage for a single VM (or even from multiple VMs on that one host). It doesn't work that way, but MPIO, when properly configured, does. However, this clearly isn't what you are trying to do. If I read your original question correctly, you have two 1Gb NICs in the storage box, and one 1Gb NIC in each of the XenServer Hosts (for storage, at least--let's ignore the other network connectivity for the moment). What you want is for each of the hosts to be able to saturate their connection to the storage box simultaneously. LACP on the storage box is exactly the correct solution here (no need for LACP on the XenServer hosts, since they only have one NIC each).
If you are really insistent on making this work with MPIO, it can be done, but would be a terrible dirty hack. You'd basically have to configure each of the hosts with a dummy NIC on the other storage network, then tell XenServer to use the two NICs in an MPIO configuration. XenCenter certainly won't let you configure it that way, so you'd have to hack it from the command line. I'm not going to tell you how to do that, cause it is the wrong thing to do. It would likely break when you make any configuration changes and would almost certainly break when you upgrade XenServer.
Trust the community: configure LACP on the Storage box only, and you'll get what you want here. If you need an analogous configuration to settle your mind, think of it as installing a 2Gb NIC in the FreeNAS box. (Of course, with that said, the other solution is to add a 10Gb NIC into the FreeNAS box, and connect it to a 10Gb port on the switch that the hosts are connected to, but I'd guess that your switch doesn't have a 10Gb port on it.)