Linux – IPv6 and NAT, routing to multiple ISPs

I have done load balancing using both lartc.org and iptables methods, and I find that the iptables method is easier to understand and implement. The only downside is that you need a fairly recent iptables version to be able to use statistic module

Let's suppose a few things:

LAN: eth0: 192.168.0.1/24

ISP1: eth1: 192.168.1.1/24, gateway: 192.168.1.2/24

ISP2: eth2: 192.168.2.1/24, gateway: 192.168.2.2/24

So here is how I would do by using iptables method:

Route tables

First edit the /etc/iproute2/rt_tables to add a map between route table numbers and ISP names

...
10 ISP1
20 ISP2
...

So table 10 and 20 is for ISP1 and ISP2, respectively. I need to populate these tables with routes from main table with this code snippet (which I have taken from hxxp://linux-ip.net/html/adv-multi-internet.html)

ip route show table main | grep -Ev '^default' \
   | while read ROUTE ; do
     ip route add table ISP1 $ROUTE
done

And add default gateway to ISP1 through that ISP1's gateway:

ip route add default via 192.168.1.2 table ISP1

Do the same for ISP2

So now I have 2 route tables, 1 for each ISP.

Iptables

OK now I use iptables to evenly distribute packets to each route tables. More info on how this work can be found here (http://www.diegolima.org/wordpress/?p=36) and here (http://home.regit.org/?page_id=7)

# iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
# iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT
# iptables -t mangle -A PREROUTING -j MARK --set-mark 10
# iptables -t mangle -A PREROUTING -m statistic --mode random --probability 0.5 -j MARK --set-mark 20
# iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

NAT

Well NAT is easy:

# iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
# iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE

First and foremost, there is nothing to fear from being on a public IP allocation, so long as your security devices are configured right.

What should I be replacing NAT with, if we don't have physically separate networks?

The same thing we've been physically separating them with since the 1980's, routers and firewalls. The one big security gain you get with NAT is that it forces you into a default-deny configuration. In order to get any service through it, you have to explicitly punch holes. The fancier devices even allow you to apply IP-based ACLs to those holes, just like a firewall. Probably because they have 'Firewall' on the box, actually.

A correctly configured firewall provides exactly the same service as a NAT gateway. NAT gateways are frequently used because they're easier to get into a secure config than most firewalls.

I hear that IPv6 and IPSEC are supposed to make all this secure somehow, but without physically separated networks that make these devices invisible to the Internet, I really can't see how.

This is a misconception. I work for a University that has a /16 IPv4 allocation, and the vast, vast majority of our IP address consumption is on that public allocation. Certainly all of our end-user workstations and printers. Our RFC1918 consumption is limited to network devices and certain specific servers where such addresses are required. I would not be surprised if you just shivered just now, because I certainly did when I showed up on my first day and saw the post-it on my monitor with my IP address.

And yet, we survive. Why? Because we have an exterior firewall configured for default-deny with limited ICMP throughput. Just because 140.160.123.45 is theoretically routeable, does not mean you can get there from wherever you are on the public internet. This is what firewalls were designed to do.

Given the right router configs, and different subnets in our allocation can be completely unreachable from each other. You do can do this in router tables or firewalls. This is a separate network and has satisfied our security auditors in the past.

There's no way in hell I'll put our billing database (With lots of credit card information!) on the internet for everyone to see.

Our billing database is on a public IPv4 address, and has been for its entire existence, but we have proof you can't get there from here. Just because an address is on the public v4 routeable list does not mean it is guaranteed to be delivered. The two firewalls between the evils of the Internet and the actual database ports filter out the evil. Even from my desk, behind the first firewall, I can't get to that database.

Credit-card information is one special case. That's subject to the PCI-DSS standards, and the standards state directly that servers that contain such data have to be behind a NAT gateway¹. Ours are, and these three servers represent our total server usage of RFC1918 addresses. It doesn't add any security, just a layer of complexity, but we need to get that checkbox checked for audits.

The original "IPv6 makes NAT a thing of the past" idea was put forward before the Internet boom really hit full mainstream. In 1995 NAT was a workaround for getting around a small IP allocation. In 2005 it was enshrined in many Security Best Practices document, and at least one major standard (PCI-DSS to be specific). The only concrete benefit NAT gives is that an external entity performing recon on the network doesn't know what the IP landscape looks like behind the NAT device (though thanks to RFC1918 they have a good guess), and on NAT-free IPv4 (such as my work) that isn't the case. It's a small step in defense-in-depth, not a big one.

The replacement for RFC1918 addresses are what are called Unique Local Addresses. Like RFC1918, they don't route unless peers specifically agree to let them route. Unlike RFC1918, they are (probably) globally unique. IPv6 address translators that translate a ULA to a Global IP do exist in the higher range perimeter gear, definitely not in the SOHO gear yet.

You can survive just fine with a public IP address. Just keep in mind that 'public' does not guarantee 'reachable', and you'll be fine.

2017 update

In the past few months, Amazon aws has been adding IPv6 support. It has just been added to their amazon-vpc offering, and their implementation gives some clues as to how large scale deployments are expected to be done.

• You are given a /56 allocation (256 subnets).
• The allocation is a fully routeable subnet.
• You are expected to set your firewall-rules (security-groups) appropriately restrictive.
• There is no NAT, it's not even offered, so all outbound traffic will come from the actual IP address of the instance.

To add one of the security benefits of NAT back in, they are now offering an Egress-only Internet Gateway. This offers one NAT-like benefit:

• Subnets behind it can't be directly accessed from the internet.

Which provides a layer of defense-in-depth, in case a misconfigred firewall rule accidentally allows inbound traffic.

This offering does not translate the internal address into a single address the way NAT does. Outbound traffic will still have the source IP of the instance that opened the connection. Firewall operators looking to whitelist resources in the VPC will be better off whitelisting netblocks, rather than specific IP addresses.

Routeable does not always mean reachable.

¹: The PCI-DSS standards changed in October 2010, the statement mandating RFC1918 addresses was removed, and 'network isolation' replaced it.