Linux – Two Node Libvirt Cluster with vxlan

kvm-virtualizationlinuxnetworking

I have two physical Ubuntu nodes, NodeA and NodeB.

On each node, I would like host libvirt guests within their own dedicated subnet. NodeA's guest network is 192.168.1.0/24, and NodeB's guest network is 192.168.2.0/24.

I need guests in NodeA be able to ping/reach guests in NodeB, and viceversa.

I have a vxlan interface on each node, 172.16.1.1/24 is assigned to the vxlan interface on NodeA, and 172.16.1.2/24 is assigned to the vxlan interface on NodeB. NodeA's vxlan address is pingable from NodeB, and vicesersa.

How would I configure routes so that guests from each node can ping eachother?
Would it be possible to use libvirt's built-in dhcp feature on each guest network?

Best Answer

A simple configuration looks possible using a 'routed' virtual network on both sides to hold guest interfaces. (In case they are currently in a NATed one, you would have to add some port forwarding, which is usually possible but may be complex if many ports are used).

Assuming the network interfaces are on 'routed' network, you just need to add the routes on both sides:

NodeA:

ip route add 192.168.2.0/24 via 172.16.1.2/24 dev eth0

(replacing eth0 with your interface name in nodeA)

NodeB:

ip route add 192.168.1.0/24 via 172.16.1.1/24 dev eth0

(replacing eth0 with your interface name in nodeA)

Related Solutions

Linux – Routing table with two NIC adapters in libvirt/KVM

So the issue was really in the NAT. I changed my default route to be on the second (bridged) interface and it worked. Btw if you really need two default routes, this is possible:

http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/

Linux – Jumbo frames between KVM guest and host

While this was an MTU problem, it turns out that it had nothing to do with the MTU settings on any of the component devices. As I showed in the original question, the host bridge, host tun interface, and guest interface all had the same MTU setting (9000 bytes).

The actual problem was a libvirt/kvm configuration issue. By default, libvirt does not use virtio devices. Absent an explicit configuration you end up with a RealTek RTL-8139 NIC. This virtual NIC does not support jumbo frames.

To use virtio devices, you need to specify an explicit model. When using virt-install:

virt-install ... -w bridge=br1,model=virtio

Or after the fact by adding a <model> tag to the appropriate <interface> element in the domain XML:

<interface type="bridge">
  <model type="virtio"/>
  <source bridge="br1"/>
  <target dev="vnet2"/>
</interface>

With this change in place, everything works as intended.

Best Answer

Related Solutions

Linux – Routing table with two NIC adapters in libvirt/KVM

Linux – Jumbo frames between KVM guest and host

Related Topic