Ubuntu – Creating a virtual network for KVM guests spanning multiple host machines

kvm-virtualizationnetworkingUbuntuubuntu-14.04virtual-machines

I have 5 physical machines running Ubuntu 14.04, all of them are together in a network. These all have static IP addresses. Each of these machines has 1 to 4 virtual machines via KVM/QEMU running Ubuntu 14.04. I'm allowed to use DHCP to get a non-static address for all the virtual machines on the physical network. But for my distributed software, I need all the virtual machines to have static IP addresses within the virtual network (as I cannot get more on the physical network). This virtual network should however span all the guests on all the virtual machines. Also, one of the virtual machines should also have a static address on the physical network.

I have managed to create a virtual network in a single host for all the virtual machines, so these can have static IPs there, but if I do this per host, it won't work. I have also created a bridged interface for the static IP on the physical network on one of the virtual machines.

This is my desired scheme:

123.45.144.0/28 (physical network)
192.168.0.0/16 (virtual network for all the vms)

123.45.147.3 (host)
    192.168.122.101, 123.45.147.49 (vm)
    192.168.122.102 (vm)
    192.168.122.103 (vm)
    192.168.122.104 (vm)

123.45.147.4 (host)
    192.168.122.105 (vm)

123.45.147.5 (host)
    192.168.122.106 (vm)

123.45.147.6 (host)
    192.168.122.107 (vm)
    192.168.122.108 (vm)

123.45.147.7 (host)
    192.168.122.109 (vm)
    192.168.122.110 (vm)

Here are outputs of ifconfig:

One of the hosts:

eth0      Link encap:Ethernet  HWaddr 5c:f9:dd:75:07:72  
          inet addr:123.45.147.3  Bcast:123.45.151.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

virbr0    Link encap:Ethernet  HWaddr 52:54:00:bd:2b:39  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Virtual machine that has both bridged interface (eth1) and an interface for the virtual network (eth0)

eth0      Link encap:Ethernet  HWaddr 52:54:00:59:2d:c8  
          inet addr:192.168.122.101  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth1      Link encap:Ethernet  HWaddr 52:54:00:4d:d0:59  
          inet addr:123.45.147.49  Bcast:123.45.151.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

After hours of googling, I was unable to find a similar setup of the problem I'm facing. As my familiarity with networks is rather low, I'd be appreciative for any leads to a solution.

Best Answer

If you have a separate ethX interface on the system with the connection to the physical network, create a second bridge on that interface. You can then configure the physical address on that virtual bridge.

The static address can be configured on the virtual machine. This makes keeping a stable address space simpler.

Related Topic