I have a machine with two interfaces that have different routeable IPv4 addresses. To return traffic on the right interface, I used this answer and comment, and it works: I can ssh into the machine using either IP address. (I don't know if this has any impact, but it might have.)
Now I added the following iptables rules for reverse proxying / DNAT using this answer. My boot script now looks like:
# eth0 is automatically brought up and gets 192.168.1.2 as IP
sudo ip link set eth1 up
sudo ip addr add 192.168.2.2/24 dev eth1
sudo ip rule add from 192.168.2.2 table frometh1
sudo ip route add default via 192.168.2.1 dev eth1 table frometh1
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.1:443
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.1:443
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
Initially, the -i eth*
was omitted and the DNAT rule was just one command. The MASQUERADE rules were also one command by exchanging -o eth*
for -d 10.0.0.1
. I split them in an attempt to make the interfaces explicit, in case iptables implicitly takes one of the interfaces.
When I try to reach 192.168.1.2:443
, the traffic is forwarded and masqueraded as expected. The hit counter for the correct DNAT rule in the PREROUTING table is incremented and the right MASQUERADE rule is incremented. I can successfully build a TCP connection.
When trying to reach 192.168.2.2:443
, I can see the traffic incoming on the interface, and the hit counter for correct DNAT rule in the PREROUTING table is incremented. But there is no outgoing packet and neither of the two MASQUERADE rules in POSTROUTING is incremented. The TCP SYN never reaches the remote system (10.0.0.1
).
Note that it does not matter if the forwarding happens through eth0 or eth1, it's fine to forward the traffic to the target (10.0.0.1
) over either eth0 or eth1.
Why do the iptables rules work only for eth0 and not for eth1?
Edit: It might be relevant to know that proxying in userspace works fine:
mkfifo /tmp/fifo
sudo nc -kvlp 443 </tmp/fifo | nc 10.0.0.1 443 >/tmp/fifo
Allows traffic to flow both ways, as described in this blog. The problem with this solution is that the remote end sees only one, persistent TCP connection. It should see one connection per client and handle multiple clients simultaneously.
Best Answer
First, let's draw your network topology.
I think this is side effect of the reverse path filter, because the packets are dropped at the routing decision step.
Check the output of
nstat -az 'TcpExtIPReversePathFilter'
command. Likely it shows nonzero counter, that are being incremented at checks.Check the routing with command
ip route get 10.0.0.1 from <external-host-ip> iif eth1
. I think it will show something likeinvalid cross-device link
.You can see the current state of
rp_filter
on the interfaces in theip netconf show
command output.Disable the
rp_filter
or better set it intoloose
mode. Therp_filter
prevents a spoofing of ip addresses in downstream networks, what is widely used in the DDoS amplification attacks. In your case the tuning doesn't add the new risks because your linux host isn't the main gateway for connected networks.It's not a last issue in your setup. Fix the problem with
rp_filter
if exists and I'll extend the answer to make your configuration work.Update
Now, when the
rp_filter
issue has resolved, let's try to make the configuration work as expected.Let's start from describing, what's happening and goes wrong.
ext2.IP
. Original packet looks likeC.IP:<someport> -> <ext2.IP>:443
GW2
and after passing through the port forwarding will looks like shown below and will be sent to the linux host.C.IP:<someport> -> 192.168.2.2:443
10.0.0.1:443
. Now we have packet in form:C.IP:<someport> -> 10.0.0.1:443
Only after the destination rewriting the linux lookups the route (the routing decision). Because the source address in the packets is still original, your additional routing rule won't involved, and the packet will be routed through the main routing table (I assume the
GW1
will be used). We'll improve it because the root cause of the issue is in this.Packet is sent to the
GW1
througheth0
interface. The source address will be rewritten by the-o eth0 -j MASQUERADE
rule. The packet will be transformed into:192.168.1.2:<otherport> -> 10.0.0.1:443
with the destination MAC address of theGW1
.GW1
forwards this packet into external network with additional source address rewriting.<ext1.IP>:<otherport2> -> 10.0.0.1:443
SYN-ACK
. The answer will look like:10.0.0.1:443 -> <ext1.IP>:<otherport2>
GW1
receives the answer, makes the reverse address translation of the destination address and sends the answer further to the linux host:10.0.0.1:443 -> 192.168.1.2:<otherport>
SYN-ACK
answer, makes the reverse address translation too and trying to lookup the route to forward the answer further. But after this step something goes wrong, but this is just a consequence of issue on the step 4. For better understanding need to check the firewall rule set.10.0.0.1:443 -> <C.IP>:<someport>
The solution
The main goal is to route back the packets through the same interface, on which these packets has been received. To do it we will use the simple routing rules.
All packets, received on the particular interface, will be routed by limited routing table, in which there are only routes for single interface. It simplest solution, but it removes the ability of the linux host forward the packets between interfaces (from
eth0
toeth1
and vice versa). If that isn't suitable for you, there is another way, but it's more complex. I'll describe it if you need.