Xen – Alternatives to Bridging Infiniband IPOIB

infinibandUbuntuxen

I've installed xen-hypervisor-4.1 package onto an Ubuntu 12.04.3 server.

It has an infiniband adaptor installed and I run IPoIB with it.

What I'm wanting to do is provide the ipoib to one or more guests. However using linux bridge is not possible.

# brctl addbr br0
# brctl addif br0 ib0
can't add ib0 to bridge br0: Invalid argument

And that's because ib0 is not a true ethernet interface. I already have ipoib deployed on the network so eoib may not be a compatible option?

Also, if I was to simply pass through the Infiniband adaptor that would only work for a single guest.

How do I make the infiniband network available to certain virtual machines?

Best Answer

IPoIB doesn't work for L2 bridging, as it's L3 over IB encapsulation. You can use eIPoIB instead by Mellanox: * http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

It's a new feature in version 2.0-2.0.51 as specified in the Release Notes. More details on eIPoIB here: * https://www.openfabrics.org/ofa-documents/doc_download/510-ethernet-tunneling-over-ipoib.html

Related Solutions

Xen domainU hosts not bridging correctly (Always 192.168.1.122)

Found the solution, it turned out we had some interference from Qemu. Observe the following file:

# cat /etc/libvirt/qemu/networks/autostart/default.xml 
<network>
  <name>default</name>
  <bridge name="virbr0" />
  <forward/>
  <ip address="192.168.122.1" netmask="255.255.255.0">
    <dhcp>
      <range start="192.168.122.2" end="192.168.122.254" />
    </dhcp>
  </ip>
</network>

This was messing with our ability to create a bridged network, and instead was forcing NAT instead. The fix was simple - remove the file and reboot! Following this our interfaces look like this:

eth0      Link encap:Ethernet  HWaddr 54:04:a6:19:25:77  
          inet addr:192.168.0.107  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::5604:a6ff:fe19:2577/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11544 errors:0 dropped:0 overruns:0 frame:0
          TX packets:316 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1693851 (1.6 MB)  TX bytes:37859 (37.8 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

peth0     Link encap:Ethernet  HWaddr 54:04:a6:19:25:77  
          inet6 addr: fe80::5604:a6ff:fe19:2577/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:16811 errors:0 dropped:0 overruns:0 frame:0
          TX packets:441 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2309354 (2.3 MB)  TX bytes:52524 (52.5 KB)
          Interrupt:18 Memory:fb500000-fb520000 

vif2.0    Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:49 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11937 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:5850 (5.8 KB)  TX bytes:1624520 (1.6 MB)

And also our bridge was automatically set up as below:

# brctl show
bridge name bridge id       STP enabled interfaces
eth0        8000.5404a6192577   no      peth0
                                    vif2.0

And so now all DomainU hosts work as intended. I hope this helps someone else :)

Linux – Xen: can’t get Dom0 and the DomUs to ping eachother

Oh my, so old... But since I got here while trying to find a solution, here's one. I think your problem is in "ip=192.168.1.2" at your vif line, but if I'm wrong, the answer is useless. And even if I'm right, it's useless for people with even slightly different config. Therefore, long version:

DomU network config (everything else doesn't matter): vif=['bridge=xenbr0']

Bridge is configured over the dummy interface, made up like this (these are commands executed as root):

# modprobe dummy
# ifconfig dummy0 up
# brctl addbr xenbr0
# brctl addif xenbr0 dummy0

Here's how it looks after you start your VM (you can see VM's vif16.0 attached to your bridge):

# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.1a088b6b402f       no              dummy0
                                                        vif16.0

I use DHCP, but you could assign IP's manually. To xenbr0 for Dom0 and eth0 under DomU for DomU. Do not assign any IPs to your bridge members! This is important.

ifconfig for Dom0 looks like this (removed stats to make it less bulky):

dummy0    Link encap:Ethernet  HWaddr 1a:08:8b:6b:40:2f
          inet6 addr: fe80::1808:8bff:fe6b:402f/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

vif16.0   Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

xenbr0    Link encap:Ethernet  HWaddr 1a:08:8b:6b:40:2f
          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
          inet6 addr: fe80::1808:8bff:fe6b:402f/64 Scope:Link

And from DomU looks like this:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.2.21  netmask 255.255.255.0  broadcast 192.168.2.255
    inet6 fe80::216:3eff:fe25:e843  prefixlen 64  scopeid 0x20<link>

Pinging Dom0:

PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.238 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.279 ms

And for internet access (my configuration is essentially NAT, yours should have internet access natively) enabling NAT may be required like this:

iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE

For troubleshooting use tcpdump -v -i xenbr0. It helps a lot as successful Ping requires your packet to get out and return, therefore gives you no clue about when it is lost - while getting out or coming back.

Best Answer

Related Solutions

Xen domainU hosts not bridging correctly (Always 192.168.1.122)

Linux – Xen: can’t get Dom0 and the DomUs to ping eachother

Related Topic