Linux – Load Balancing and NAT-ing Multiple ISP Connections

linuxload balancinglocal-area-networknat;networking

I have two internet connections from two different ISPs and I need to balance the traffic originating to and from my network between the two ISPs. I am using Debian GNU/Linux.

My set-up is like this –

eth0 (192.168.0.0/24) — Local network

eth1 (192.168.1.0/24) — ISP #1

eth2 (192.168.2.0/24) — ISP #2

My local network is connected to this server via eth0 and the box is the DHCP server cum Gateway for all the machines in the LAN.

The server needs to do load-balancing between the two ISPs and also needs to do NAT-ing.

I have followed the routing instructions on lartc.org but I still need instructions to do the NAT-ing properly.

Any help will be appreciated.

PS – I know about pFsense but I need to use Linux.

Best Answer

I have done load balancing using both lartc.org and iptables methods, and I find that the iptables method is easier to understand and implement. The only downside is that you need a fairly recent iptables version to be able to use statistic module

Let's suppose a few things:

LAN: eth0: 192.168.0.1/24

ISP1: eth1: 192.168.1.1/24, gateway: 192.168.1.2/24

ISP2: eth2: 192.168.2.1/24, gateway: 192.168.2.2/24

So here is how I would do by using iptables method:

Route tables

First edit the /etc/iproute2/rt_tables to add a map between route table numbers and ISP names

...
10 ISP1
20 ISP2
...

So table 10 and 20 is for ISP1 and ISP2, respectively. I need to populate these tables with routes from main table with this code snippet (which I have taken from hxxp://linux-ip.net/html/adv-multi-internet.html)

ip route show table main | grep -Ev '^default' \
   | while read ROUTE ; do
     ip route add table ISP1 $ROUTE
done

And add default gateway to ISP1 through that ISP1's gateway:

ip route add default via 192.168.1.2 table ISP1

Do the same for ISP2

So now I have 2 route tables, 1 for each ISP.

Iptables

OK now I use iptables to evenly distribute packets to each route tables. More info on how this work can be found here (http://www.diegolima.org/wordpress/?p=36) and here (http://home.regit.org/?page_id=7)

# iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
# iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
# iptables -t mangle -A PREROUTING -j MARK --set-mark 10
# iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
# iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

NAT

Well NAT is easy:

# iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

Related Solutions

Firewall – Multiple interface NAT routing

These rules probably don't do what you want. The first rule permits any new connection from eth1 and eth2, the second permits everything from eth0 and eth1. This appears to permit everything from any interface basically making your firewall pointless.

iptables -A INPUT -m state --state NEW -i ! eth0 -j ACCEPT
iptables -A INPUT -m state --state NEW -i ! eth2 -j ACCEPT

You may mean this instead.

iptables -A INPUT -m state --state NEW -i eth1 -j ACCEPT

Anyway, instead of adding things to rc.local I would probably update your /etc/network/interfaces to look like this. I am making some guesses about what you have for masks, so be sure to check and update. You probably should be adding all of the link routes to the fiber table.

auto eth0
iface eth0 inet static
  address 192.168.1.2
  netmask 255.255.255.0
  network 192.168.1.0
  broadcast 192.168.1.255
  gateway 192.168.1.1
  up ip route add table fiber scope link proto kernel dev eth0 192.168.1.0/24

auto eth1
iface eth1 inet static
  address 10.254.239.1
  netmask 255.255.255.0
  network 10.254.239.0
  broadcast 10.254.239.255
  up ip route add table fiber scope link proto kernel dev eth1 10.254.239.0/24

auto eth2
iface eth2 inet static
  address 20.20.20.22
  netmask 255.255.255.252
  network 20.20.20.20
  broadcast 20.20.20.23
  up ip route add table fiber scope link proto kernel dev eth2 20.20.20.20/30
  up ip route add default via 20.20.20.21 table fiber
  up ip rule add fwmark 2 table fiber

Ubuntu – Load balancing ISPs on Ubuntu Server

I believe the script you found was created by following/amalgamating the various pieces of information from the Linux Advanced Routing & Traffic Control HOWTO. Specifically the section on load balancing separate ISP links. I suggest you read that page to help understand what that script is doing. Keep in mind that that How To references old Kernels though I don't think the principle would have changed much and the ip commands are probably still valid.

That script would need to be run each time the machine boots to set the routes etc. so you don't need to backup files (it doesn't touch any). Once you reboot the machine the rules will be forgotten and the machine's default network setup will be used.

If you don't want to reboot then, depending on the distro, you may be able to restart the networking i.e. on Red Hat a service network restart should clear the routes for you and add the default route again (this is based on my quick scan of the network scripts). I believe Ubuntu will have a similar system (I haven't used it for some time and they use Upstart nowadays which I have no knowledge of). Otherwise you'd have to manually remove the rules that that script creates (or write a script that does it) using the ip command.