Linux – two subnets, one interface, one gateway – NAT the additional subnet

iptableslinuxnat;networkingrouter

We currently have a few servers that all use the same LAN:

Host1: eth0 10.0.0.1/24
Host2: eth0 10.0.0.2/24
Host3: eth0 10.0.0.3/24
Gateway: 10.0.0.254

We want to run some VMs (VirtualBox) on these servers. We can set them up to bridge onto the host's eth0, but we cannot use addresses from the 10.0.0.0/24 range since they may be allocated in future.

So we figured we'd use a different subnet:

Host1VM: eth0 192.168.0.1/24 (bridge to host eth0)
Host2VM: eth0 192.168.0.2/24 (bridge to host eth0)
Host3VM: eth0 192.168.0.3/24 (bridge to host eth0)

That's fine and all the VMs can communicate with each other since they're on the same subnet which uses the same physical interface.

The issue we face is we need to give those VMs access to the internet via the 10.0.0.254 gateway. So we figured why not pick one of the hosts and use it as a router/NAT?

Host1: eth0 10.0.0.1/24, eth0:0 192.168.0.254/24

Now we can give the VMs a gateway of 192.168.0.254. The problem we then see is that Host1 doesn't seem to NAT properly.

iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o eth0 -j SNAT --to 10.0.0.1

I thought that would work, and we do see it matching packets. If a VM pings the internet, we see the ICMP packet come IN to Host1 (since it's the router), and then the host re-sends the ICMP because it's NAT'ing, the internet host responds back to the host – but then it dies there. I expected the host to then forward the packet back to the VM, but it doesn't.

What am I missing, or is this setup simply not possible?

Edit: Just to clarify, we have no DENY rules within iptables, everything is default ACCEPT. We have also enable IP forwarding.

Update1 – iptables

Ignore the virbr0 – that's not related to VirtualBox VMs

# Completed on Fri Sep 20 16:50:45 2013
# Generated by iptables-save v1.4.12 on Fri Sep 20 16:50:45 2013
*nat
:PREROUTING ACCEPT [171383:10358740]
:INPUT ACCEPT [1923:115365]
:OUTPUT ACCEPT [192:21531]
:POSTROUTING ACCEPT [169544:10254463]
-A POSTROUTING -s 192.168.0.0/24 -o eth0 -j SNAT --to-source 10.0.0.1
COMMIT
# Completed on Fri Sep 20 16:50:45 2013
# Generated by iptables-save v1.4.12 on Fri Sep 20 16:50:45 2013
*filter
:INPUT ACCEPT [96628707:25146145432]
:FORWARD ACCEPT [195035595:22524430122]
:OUTPUT ACCEPT [44035412:304951330498]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Fri Sep 20 16:50:45 2013
# Generated by iptables-save v1.4.12 on Fri Sep 20 16:50:45 2013
*mangle
:PREROUTING ACCEPT [291641356:47665886851]
:INPUT ACCEPT [96628707:25146145432]
:FORWARD ACCEPT [195035595:22524430122]
:OUTPUT ACCEPT [44035838:304951365412]
:POSTROUTING ACCEPT [239078922:327477732680]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Fri Sep 20 16:50:45 2013

Update 2 – tcpdump

16:58:37.189758 IP 192.168.0.2 > 74.125.128.106: ICMP echo request, id 1, seq 2, length 40
16:58:37.189805 IP 10.0.0.1 > 74.125.128.106: ICMP echo request, id 1, seq 2, length 40
16:58:37.194607 IP 74.125.128.106 > 10.0.0.1: ICMP echo reply, id 1, seq 2, length 40
(no final reply back to the VM)

Best Answer

This is telling:

16:58:37.189758 IP 192.168.0.2 > 74.125.128.106: ICMP echo request, id 1, seq 2, length 40
16:58:37.189805 IP 10.0.0.1 > 74.125.128.106: ICMP echo request, id 1, seq 2, length 40
16:58:37.194607 IP 74.125.128.106 > 10.0.0.1: ICMP echo reply, id 1, seq 2, length 40

The upstream router (10.0.0.254) is sending the reply back to Host1, so your routing and SNAT is working. The problem is that Host1 is not passing that reply back to the 192.168.0.0/24 network.

Have you got the appropriate connection tracking kernel modules loaded?

Make sure the traffic is going into the connection tracking table:

grep src=192.168.0.2 /proc/1/net/nf_conntrack

I saw a similar issue with TCP packets just the other day which turned out to be due to rp_filter in the kernel. I can't see how that would be the issue here, but it's very similar. Can you post the route table (ip route show) from Host1 and check your rp_filter setting? (cat /proc/sys/net/ipv4/conf/default/rp_filter)

Related Topic