Amazon EC2 – How to Route Traffic via Specific Interface on Gateway Node

amazon ec2gatewayiproute2iptables

I have linux ec2 instances with two nics (eth0 and eth2). Both the nics have public ip's attached to it and are able to get out to the internet. This linux instance is acting as a gateway node for me, forwarding traffic from ec2 in private subnets who dont have internet access.

Am able to forward traffic from ec2 instance in private subnet to this gateway instance in public subnet. All the traffic in the gateway instance is received on eth0. Now I would like to be able to get out to the internet via eth1 and not eth0. I want to do so because i want the outside world to see eth1's IP as the source IP.

Performing these steps helps me achieve this

ip route add 10.0.0.0/23 <subnet of gateway cidr> dev eth2 table 2
ip rule add from <private ec2 instance's ip> table 2
ip route add default via 10.0.0.1 dev eth2 table 2
ip route flush cache

iptables -A FORWARD --in-interface eth0 -j ACCEPT #accepts traffic on eth0
iptables --table nat -A POSTROUTING --out-interface eth2 -j MASQUERADE #forwards the traffic outside via eth2

With this setup, when i curl outside of ec2 instance in private subnet (via gateway node) i see what i expect to see (which is that the outside world sees the request is coming from eth1's ip), but the curl requests takes a really long time (and sometimes it even times out). I would say about 5 out of 10 requests succeed.

What am i missing here?

Thanks
Kay

Best Answer

Apparently, the order of the rules matters a lot. I was able to specify the order of the rules by doing something like this.

ip rule add preference 512 table 2

Once I got the ordering right, I was able to egress via a specific interface correctly.

Related Solutions

Iptables – Routing squid traffic via same interface over different gateway (= DSL modem)

Basically, you'll need iproute2 functionality since what you are doing is source-based routing:

echo 1 squid >> /etc/iproute2/rt_tables
ip route add default 192.168.50.250 table squid
ip rule add from 192.168.50.1 lookup squid

Apparently the ipcop kernel is built without the FWMARK featureset, so this won't work for you. There also has been a "ROUTE" target for iptables once in patch-o-matic which could do the same, but I believe its development has been abandoned as the behavior is redundant and the functionality is inferior to iproute2. I also don't think it has been included with recent ipcop's releases - you would have to compile it itself and add it to your ipcop installation.

A quick search turned up a precompiled version for an outdated 1.4.15 version of ipcop (probably won't work with more recent versions, but you can give it a try) - there might be more recent builds, you should ask on the ipcop mailing lists or forums if you don't want to compile it yourself.

Linux – Selecting gateway on application level on Linux

I can think of one way to do it which is as follows.

Define two IP addresses on your NIC like 10.0.0.1 & 10.0.0.2. This should be easy and straight-forward. This can be done using ifconfig.
Configure source based routing such that each source IP is routed to one of the default gateways. This link can help you.
Finally, you can tell your application to use bind one of the addresses 10.0.0.1 or 10.0.0.2. This way you can choose one of the gateways depending on the source IP selected. To test this, you can use telnet with -b source_ip option.

Here are the commands I used previously to enable source-based routing:

$ sudo ip rule add from 10.0.0.2 tab 1 priority 500
$ sudo ip route add default via 10.0.0.101 dev eth0 tab 1
$ sudo ip route flush cache

If you have the default gateway set to 10.0.0.100, then this should work for you. The packets sent from 10.0.0.1 should be sent to default gateway, and packets sent from 10.0.0.2 should be sent to 2nd gateway 10.0.0.101 as instructed above.

Related Topic