Linux Routing with two NICs (LAN vs Internet) with NAT and bridging for VMs

My Setup:

There is only one physical machine in this setup, a Host System for Virtual Machines (VMs) with two network adapters.

One NIC (eth0) is connected to an internal network (LAN subnet, e.g. 10.x.x.x/24) and shall be used for internal traffic.

The other NIC (eth1) is connected to public internet (it has a public routable IP configured). This connection shall be used to port-forward public internet traffic to internal IPs of the VMs (incoming traffic) and to allow the VMs to access public internet (outgoing traffic) via NAT.

Virtual Machines use IP addresses in the LAN-Subnet (10.x.x.x/24, same as eth0)

I've got a bridge device (br0) configured for virtual network interfaces of the VMs (vnet0, vnet1, …) and the LAN-NIC (eth0). That means:

• br0 has an IP-Adress in the LAN subnet (10.x.x.x/24)
• eth0 is added to the bridge
• vnet0, vnet1, … (used by the VMs) are dynamically added to the bridge

Problems

Communication within the LAN works fine. Also the VM-Host is accessable via the public IP and has internet access.

My problem is the NAT configuration to allow the VMs to access public internet, too.

I tried to use a simple (S)NAT rule:

iptables -t nat -I POSTROUTING -s 10.x.x.x/24 ! -d 10.x.x.x/24 -j SNAT --to-source y.y.y.102

Whereas y.y.y.102 is the public routable IP of the second NIC (eth1).

I found out that I need to enable "ip_forward" and "bridge-nf-call-iptables":

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

Else the bridged packages won't be processed by iptables.

Now the packets from the VMs seem to go through the following Chains of iptables:

• "FORWARD" (regular) – I accept them there (-j ACCEPT, counter goes up)
• "PREROUTING" (nat) – I accept them there (policy ACCEPT, counter goes up)
• "POSTROUTING" (nat) – They match the SNAT rule

But not all packets seem to arrive at PRE/POSTROUTING for any reason I couldn't figure out so far.

However, more interestingly tcpdump -i eth0 vs. tcpdump -i eth1 show that the packets (I tried to ping an external IP from within a VM) seem to be sent via the wrong interface eth0 (=LAN-NIC). Even the NAT rule was applied, so the source address was changed to the IP of the other NIC (eth1).

QUESTIONs:

How can I configure the system to output the NATed packets with the public IP as source address to be sent over the correct NIC (eth1)?

Do I somehow need to add eth1 to the bridge (br0)? If so, how do I assign the public IP address correctly? Usually the IP needs to be configured on the bridge device. Would I need to assign an alias adress to the bridge (public IP on br0:0)?

Configuration Details

The routing configuration on the host system:

# ip r
default via y.y.y.126 dev eth1
10.x.x.0/24 dev br0  proto kernel  scope link  src 10.x.x.11
y.y.y.96/27 dev eth1 proto kernel  scope link  src y.y.y.102

• IP: y.y.y.126 is our router for public internet.
• IP: y.y.y.102 is the public IP of the host machine
• IP: 10.x.x.11 is the LAN IP of the host machine
• SUBNET: 10.x.x.0/24 is the LAN
• SUBNET: y.y.y.96/27 is the public IP subnet

NIC configuration:

# ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.x.x.11  netmask 255.255.255.0  broadcast 10.x.x.255
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 0  (Ethernet)
        RX packets 2139490  bytes 243693436 (232.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29085  bytes 2398024 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 1000  (Ethernet)
        RX packets 2521995  bytes 290600491 (277.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 383089  bytes 48876399 (46.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdfa60000-dfa7ffff

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet y.y.y.102  netmask 255.255.255.224  broadcast y.y.y.127
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 1000  (Ethernet)
        RX packets 2681476  bytes 597532550 (569.8 MiB)
        RX errors 0  dropped 130  overruns 0  frame 0
        TX packets 187755  bytes 21894113 (20.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdfa00000-dfa1ffff

Bridge configuration:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.002590eb1900       no              eth0
                                                        vnet0

And iptables rules:

# iptables -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  723  106K DROP       udp  --  *      *       y.y.y.0/24           0.0.0.0/0            udp spt:5404
  586 40052 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    5   420 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    2   458 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4
    2   458 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1343  173K ACCEPT     tcp  --  *      *       10.x.x.2             0.0.0.0/0            tcp spt:3389
 1648  127K ACCEPT     tcp  --  *      *       0.0.0.0/0            10.x.x.2             tcp dpt:3389
   18  1040 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4
   18  1040 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 525 packets, 84016 bytes)
 pkts bytes target     prot opt in     out     source               destination


# iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 13 packets, 1218 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 5 packets, 420 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 13 packets, 880 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 14 packets, 920 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   300 SNAT       all  --  *      *       10.x.x.0/24          !10.x.x.0/24           to:y.y.y.102

And here a captured NATed packet (ping from VM) on LAN interface card:

# tcpdump -i eth0
12:53:55.243350 IP y.y.y.102 > y.y.y.110: ICMP echo request, id 2, seq 5, length 40

Output of "ip rule":

# ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default