I just have a problem on the multiple NIC & IP addresses server. I've configured my Cent OS 7 server with an additional public IP address which belongs to eth1. When iptables source NAT enabled, only the additional IP address couldn't be used.
In this post, two public IP addresses are indicated by 99.0.1.100 (eth0) and 99.0.2.100 (eth1). Subnet-mask of 99.0.2.100 is 255.255.254.0, default gateway is set properly (99.0.1.1 for 99.0.1.100, 99.0.2.1 for 99.0.2.100).
below is contents of /etc/sysconfig/network-scripts/ifcfg-eth1
. (HWADDR is censored)
DEVICE=eth1
TYPE=Ethernet
HWADDR=**:**:**:**:**:**
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=99.0.2.100
NETMASK=255.255.254.0
below is output of ip route
.
default via 99.0.1.1 dev eth0 proto static metric 100
99.0.1.0/23 dev eth0 proto kernel scope link src 99.0.1.100 metric 100
99.0.2.0/23 dev eth1 proto kernel scope link src 99.0.2.100
below is output of ip rule
.
0: from all lookup local
32765: from 99.0.2.100 lookup subroute-eth1
32766: from all lookup main
32767: from all lookup default
below is contents of /etc/sysconfig/network-scripts/route-eth1
.
default via 99.0.2.1 table subroute-eth1
below is contents of /etc/sysconfig/network-scripts/rule-eth1
.
from 99.0.2.100 table subroute-eth1
then, the following request has been sent properly and responded proper IP address as string for each public IP addresses.
# curl --interface eth0 api.ipify.org # outputs 99.0.1.100
# curl --interface eth1 api.ipify.org # outputs 99.0.2.100
here, I want each users of the system use different public IP address. So applied the following iptables source NAT. Where uid=1000 is A, uid=1001 is B, respectively.
# iptables -t nat -m owner --uid-owner 1000 -A POSTROUTING -j SNAT --to-source 99.0.1.100
# iptables -t nat -m owner --uid-owner 1001 -A POSTROUTING -j SNAT --to-source 99.0.2.100
in this situation, eth0 works correctly, but eth1 has timed out.
# curl api.ipify.org # outputs 99.0.1.100
# sudo -u A curl api.ipify.org # outputs 99.0.1.100
# sudo -u B curl api.ipify.org # **time out**
I have no idea what is wrong because of lack of network experience. If anyone has ideas, help me, please.
Note that in this situation, ssh root@99.0.2.100
from Internet is still working properly.
Edit: below is output of ip route show table subroute-eth1
default via 99.0.2.1 dev eth1
Best Answer
The uid should trigger a routing change to use
eth1
instead ofeth0
.nat/POSTROUTING
's SNAT (while still needed) comes too late for this because as the name POSTROUTING implies it happens after the routing decisions: the network interface is chosen and won't change anymore.For local traffic the rule has to happen in the
mangle/OUTPUT
chain to alter the route, by using marks which can trigger anip rule
lookup:The information is then reused via
ip rule
. Even 1000 has to be considered (to handle the specific caseuid1000$ curl --interface eth1 http://api.ipify.org/
which should not use entry 32765):Note that the initial default IP was chosen before: SNAT is still needed to fix it to the new correct one. Why using this mark since there's already a rule with
from 99.0.2.100 lookup subroute-eth1
then? Again that's because of the order of various evaluations in the network stack, as summarized by this Packet flow in Netfilter and General Networking schematic: only a change inmangle/OUTPUT
can trigger the reroute check seen in the schematic, and by this time the IP hasn't been corrected to 99.0.2.100.In the reverse path, some network elements will not know about those routing tables in time anyway, loose mode has to be set on
eth1
's rp_filter or return packets will be dropped:Same for
eth0
to handle the specific case for uid 1000 above (uid1000$ curl --interface eth1 http://api.ipify.org/
):With this it should mostly work. But actually a few packets aren't owned by uid 1001:
TCP RST
, lastACK
once the process has ended, or for UDP, related ICMP errors... are owned only by the kernel. They won't match uid 1001, will be sent on the wrong interface (with the wrong IP on this wrong interface) and this will create some timeouts at the end of the connections, more probably at the remote side rather than locally. This can be seen by doing akill -KILL
on a client with an altered connection made with user 1001 viaeth1
, and witnessing several retryACK
packets with source 99.0.2.100 sent viaeth0
(instead ofeth1
) acknowledging theFIN
from remote received several times oneth1
.So all this has to be reworked using
CONNMARK
to track flows initiated by the user, instead of packets initiated by the user (some explanations here).nat
doesn't have to change, since only the 1st packet inNEW
state goes through iptables rules anyway and it's tracking flows already. Let's start over:Everything should behave correctly now.
Final note: while it wasn't needed in your setting, the link local route for eth1 should be duplicated on the table
subroute-eth1
, else accessingeth1
's LAN from uid 1001 will give strange results, because rule 32765 is read before rule 32766 so there won't ever be the link local route defined, as the result of this command shows:It might prevent access to all of the LAN except the gateway.
You should add (or put as appropriate in the system configuration files):
And then the previous
ip route get
command will give instead the correct: