How to route all VM traffic through specific physical interface over a Linux bridge

kvm-virtualizationlinux-networkingvirtual-network

My objective is to have all KVM guest VMs send and receive traffic on em2 with addresses on the 192.168.2.0/24 subnet.

I have a host Linux machine (CentOS 7) with several NICs, 2 of which are in use in this scenario, em1 and em2.

The em1 interface has an IP of 192.168.0.131. The em2 interface has been attached to br0, so it doesn't have an IP itself, but br0 has been assigned an IP address of 192.168.2.1.

I have created a route on my Netgear firewall to direct 192.168.2.0/24 traffic to 192.168.2.1 but this address doesn't show as an attached device the way 192.168.0.131 does, maybe because it's a virtual Linux bridge.

From the host VM, I can ping both the "bridge gateway", the VM guest, and the firewall gateway to the internet:

[root@boss ~]# ping -c1 192.168.2.1
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.085 ms

[root@boss ~]# ping -c1 192.168.2.10
64 bytes from 192.168.2.10: icmp_seq=1 ttl=64 time=0.476 ms

[root@boss ~]# ping -c1 192.168.0.254
64 bytes from 192.168.0.254: icmp_seq=1 ttl=64 time=4.17 ms

And from the guest VM, I can ping em1, but not the internet gateway, 192.168.0.254:

[root@localhost ~]# ping -c1 192.168.0.131
64 bytes from 192.168.0.131: icmp_seq=1 ttl=64 time=0.282 ms

[root@localhost ~]# ping -c1 192.168.0.254
PING 192.168.0.254 (192.168.0.254) 56(84) bytes of data.

--- 192.168.0.254 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

This is my config for em2:

DEVICE=em2
TYPE=Ethernet
ONBOOT=yes
BRIDGE=br0

And br0:

DEVICE=br0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bridge
IPADDR=192.168.2.1
PREFIX=24
GATEWAY=192.168.0.254
ZONE=public
STP=no

My routing table on the VM host:

[root@boss ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.254   0.0.0.0         UG    0      0        0 em1
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 em1
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 br0

The guest VM was started with virt-install:

virt-install \
  --name vm-guest-1 \
  --network bridge=br0 \
  --virt-type kvm \

Guest VM eth0:

DEVICE="eth0"
BOOTPROTO="none"
ONBOOT="yes"
TYPE="Ethernet"
IPADDR="192.168.2.10"
NETMASK=255.255.255.0

And the guest VM routing table:

[root@localhost ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 eth0
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

As requested, my host bridge output:

[root@boss ~]# brctl show
bridge name     bridge id           STP enabled    interfaces
br0             8000.d4ae529de039   no             em2
                                                   vnet0

Question/Problem:

How do I / why can't I route into my guest VM, or rather, why can't my guest VM get out to the internet?

Best Answer

You have basically two options:

  1. In your current configuration vnet0 and em2 are bridged, so the VM can reach the Netgear device without routing. You just need to assign the VM an address in 192.168.0.0/24 or use DHCP.

  2. If you need to route, remove the em2 interface from the bridge and assign to it an address in 192.168.0.0/24. Then add a new routing table to /etc/iproute2/rt_tables:

    100 vms

    Now add a routing table selection rule and a default route for your VMs:

    ip route add default dev em2 via 192.168.0.254 table vms
ip rule add iif br0 table vms

    In addition you need to modify the static route on the Netgear device to point to em2's address.

Edit: In the second configuration Ethernet frames from the VMs do not leak on the physical network. The additional routing table is selected only when it needs to route something from the bridge br0 (i.e. the virtual guests) and sends the traffic through em2 instead of em1 (the default route in the main table). You can see all routing tables at once with:

ip route show table all

You can read more about routing tables on the Guide to IP Layer Network Administration with Linux.

Related Topic