KVM bridge over bonding with multiple subnets

bondingbridgecentos7kvm-virtualization

I can access host via 213.xxx.xxx.1 assigned to br0:1 but VM with 213.xxx.xxx.2 (bridge interface on br0:1) doesn't work.

Do I have to create another bridge interface for the second subnet, but then bond0 is already connected with br0?!

Basically how I can use the second subnet for VMs in this scenario?

Subnets:

213.xxx.xxx.176/28
213.xxx.xxx.0/27 routed via 213.xxx.xx.180

Setup:

em[1-2]

# cat /etc/sysconfig/network-scripts/ifcfg-em[1-2]
DEVICE="em[1-2]"
NAME="bond0-slave[1-2]"
BOOTPROTO="none"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL=no
IPV6INIT=no
PEERDNS=no
MASTER=bond0
SLAVE=yes

bond0

# cat /etc/sysconfig/network-scripts/ifcfg-bond0 
DEVICE="bond0"
NAME="bond0"
BOOTPROTO="none"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Bond"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"
BONDING_OPTS="mode=active-backup miimon=100"
BRIDGE=br0

br0

# cat /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE="br0"
NAME="br0"
BOOTPROTO="none"
IPADDR="213.xxx.xxx.180"
GATEWAY="213.xxx.xxx.177"
NETMASK="255.255.255.240"
BROADCAST="213.xxx.xxx.191"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Bridge"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"

br0:1

# cat /etc/sysconfig/network-scripts/ifcfg-br0:1 
DEVICE="br0:1"
NAME="br0:1"
BOOTPROTO="none"
IPADDR="213.xxx.xxx.1"
NETMASK="255.255.255.224"
BROADCAST="213.xxx.xxx.31"
NM_CONTROLLED="yes"
#ONBOOT="yes"
TYPE="Bridge"
USERCTL="no"
IPV6INIT="no"
PEERDNS="no"
ONPARENT="yes"

Update Wed 18 Jan 16:25:17 GMT 2017:

<interface type='bridge'>
      <mac address='52:54:00:4c:4f:27'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Update Tue 24 Jan 11:44:21 GMT 2017:

VM network setup:

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
TYPE="Ethernet"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
NAME="eth0"
DEVICE="eth0"
ONBOOT="yes"
IPADDR="213.xxx.xxx.2"
PREFIX="27"
GATEWAY="213.xxx.xxx.177" <-- not sure if this should be 213.xxx.xxx.180 instead?!

IPTables on host:

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68

# iptables -t nat -nL
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
RETURN     all  --  192.168.122.0/24     224.0.0.0/24        
RETURN     all  --  192.168.122.0/24     255.255.255.255     
MASQUERADE  tcp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  udp  --  192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
MASQUERADE  all  --  192.168.122.0/24    !192.168.122.0/24

It works when I set a default gateway to 213.xxx.xxx.1 on VM (IP from br0:1 on the host) – not sure why I can't just use the same default gateway like host (213.xxx.xxx.177).

I had to also flush IPTables so I need to find out what rules are necessary.

I'm also getting redirection when pinging from VM so not sure if this config is optimal:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 213.xxx.xxx.1: icmp_seq=1 Redirect Host(New nexthop: 213.xxx.xxx.177)
From 213.xxx.xxx.1 icmp_seq=1 Redirect Host(New nexthop: 213.xxx.xxx.177)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=7.26 ms
...

Update Tue 24 Jan 14:49:43 GMT 2017:

I'm not sure why IPTables actually filter packets traversing the bridge even if br_netfilter module is not loaded

# lsmod | grep "br_netfilter"; echo $?
1
# test -d /proc/sys/net/bridge; echo $?
1

This rule also didn't help:

# iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

Best Answer

Solution which "answers the question":

  1. Make sure ip forwarding is enabled on the host if cat /proc/sys/net/ipv4/ip_forward is not 1, look on this site how to change it
  2. Change the VM gateway to the hosts ip on the vm subnet: 213.xxx.xxx.1
  3. Fix your broken iptable configuration. iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT looks fine. Eventually change it to something link iptables -I FORWARD -s 213.xxx.xxx.0/27 -j ACCEPT ( this means: accept all traffic from the vms network to an other network )

The idea of this solution is to use your host as gateway for the vm network. Obviously if you already have an other gateway on the 213.x.x.0 network ( 213.xxx.xx.180 ) you can use it directly from your vms.

The suggested iptables configuration will allow traffic from the VMs to the outside. If you want to allow connexions from the outside to the VMs you have to put a rule looking like iptables -I FORWARD -d 213.xxx.xxx.0/27 -j ACCEPT

Recommended solution: Keep in mind that all this is probably just a mess because of the way the question is asked. In the real world, if you have a router / firewall which manages your two subnets, you should take the easy way: - the router/firewall will have an ip address on each subnet. It will be gateway for all the hosts in the subnets - assign a vlan to each subnet - on your host, you will have 2 interfaces and to bridges - your vms will be bridged on the "vms subnet" and use the firewall as gateway

This way you will have no crazy iptables / alias / forwarding configuration. You will have some hosts on one subnet, and others on the other subnet, and the routing / acls / nats configured on your firewall. You should not even assign an ip address on the "vm network" to your host.

Merging two networks on the same broadcast domain is a source of problems, and configuring multiple ip addresses on a host should be avoided if you are not very self-confident in network configuration.

Related Topic