Linux – How to load-balance traffic across two independent NAT devices using Linux kernel

iproute2iptableslinuxlinux-networking

I have a network with several Linux workstations/servers on it, and two Internet connections, each with a consumer-grade NAT router in front of it. So for concreteness, let's say the devices are:

192.0.2.0/26  network
192.0.2.1     router A
192.0.2.62    router B
192.0.2.10    host X
192.0.2.11    host Y

Here's a diagram, similar to one in the LARTC HOWTO:

                                                             ________
                                           +------------+        /
                                           |            |       |
         +-----------+-[NAT router A]------+ Provider 1 +-------
        _|                                 |            |     /
    ___/  \_         +--------------+      +------------+    |
  _/        \__      | Linux host   |                      /
 /             \     |              |                      |
| Local network -----+  if1         |                      |     Internet
 \_           __/    |              |                      |
   \__     __/       |              |                      \
      \___/          +--------------+     +------------+    |
        |                                 |            |     \
        +---------------[NAT router B]----+ Provider 2 +-------
                                          |            |       |
                                          +------------+        \________

If a new TCP connection comes in via one or the other router, return traffic needs to go out via that router. Otherwise the return SYN packet will be improperly NATted (if even passed out at all), and the connection will effectively be blackholed. I could set up mutually exclusive port-forwarding sets on the two routers if that would help, but I'd rather keep them the same (or at least in some cases overlapping) if possible.

For new outgoing connections, I want to allocate traffic by range using the main routing table, but for certain traffic I might want to randomly balance new connections across the two routers.

I suspect what I'm asking about is doable using iptables and ip, or perhaps either of those tools alone. However, the HOWTOs and other answers I've found seem to all address a single router (running Linux), or host, with multiple interfaces… not policy among multiple routers attached to the same interface.

Edit: I've found one other question, also unanswered, that's asking about the same situation (note 192.168.0.0/23 as the local network), but focuses on policy for outgoing traffic; my question here is specifically about policy for incoming traffic. (Both NAT devices have a "port forward by port-range to host" function.)

Best Answer

Yes, it's possible to do this using iptables and ip together. In my case all the Linux boxes at issue run Debian Stable, so I wrote a custom script for /etc/network/if-up.d.

# First set up two routing-tables
ip route add to default table 11 via 192.0.2.1  dev $IFACE
ip route add to default table 33 via 192.0.2.62 dev $IFACE

# ... and rules to use them.
ip rule add priority 99 table 11 fwmark 11
ip rule add priority 99 table 33 fwmark 33

# Copy connmark to fwmark for applicable outgoing packets.
iptables -t mangle -A OUTPUT ! -d 192.0.2.0/26 \
  -m addrtype --dst-type UNICAST -j CONNMARK --restore-mark

# Set connmark for applicable incoming packets.
iptables -A INPUT -i $IFACE ! -s 192.0.2.0/26 \
  -m addrtype --src-type UNICAST \
  -m mac --mac-source 00:00:5E:00:53:05 \
  -j CONNMARK --set-mark 11
iptables -A INPUT -i $IFACE ! -s 192.0.2.0/26 \
  -m addrtype --src-type UNICAST \
  -m mac --mac-source 00:00:5E:00:53:07 \
  -j CONNMARK --set-mark 33

Note that it's defined in that order so that the whole thing takes effect with the last two commands. 00:00:5E:00:53:05 and 00:00:5E:00:53:07 would be the on-link addresses of the two NAT devices.

Related Topic