Iptables – route traffic from a cgroup via a specific network interface

I've recently hit a similar issue, albeit a slightly different. I wanted to route only TCP port 25 (SMTP) over an OpenVPN tap0 interface, while routing all other traffic (even for the same host) over the default interface.

To do so, I had to mark packets and set up rules for handling it. First, add a rule that make the kernel route packets marked with 2 through table 3 (explained later):

ip rule add fwmark 2 table 3

You could have added a symbolic name to /etc/iproute2/rt_tables, but I did not bother to do so. The number 2 and 3 are randomly chosen. In fact, these can be the same but for clarity I did not do it in this example (although I do it in my own setup).

Add a route for redirecting traffic over a different interface, assuming the gateway being 10.0.0.1:

ip route add default via 10.0.0.1 table 3

Very important! Flush your routing cache, otherwise you will not get a response back and sit with your hands in your hair for some hours:

ip route flush cache

Now, set a firewall rule for marking designated packets:

iptables -t mangle -A OUTPUT -p tcp --dport 465 -j MARK --set-mark 2

The above rule applies only if the packets come from the local machine. See http://inai.de/images/nf-packet-flow.png. Adjust it to your requirements. For instance, if you only want to route outgoing HTTP traffic over the tap0 interface, change 465 to 80.

To prevent the packets sent over tap0 getting your LAN address as source IP, use the following rule to change it to your interface address (assuming 10.0.0.2 as IP address for interface tap0):

iptables -t nat -A POSTROUTING -o tap0 -j SNAT --to-source 10.0.0.2

Finally, relax the reverse path source validation. Some suggest you to set it to 0, but 2 seems a better choice according to https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt. If you skip this, you will receive packets (this can be confirmed using tcpdump -i tap0 -n), but packets do not get accepted. The command to change the setting so packets get accepted:

sysctl -w net.ipv4.conf.tap0.rp_filter=2

first add a firewall rules:

iptables -t mangle -A OUTPUT -m owner --uid USER -j MARK --set-mark 1
iptables -t nat -A POSTROUTING -m mark --mark 1 -j MASQUERADE

then add a routing rule:

ip rule add fwmark 0x1 table 100

and then add routes to your new routing table:

ip route add SOMEROUTE via SOMEGATEWAY table 100