So, I'm trying to implement per-user routing so that I'm able to route all btpd torrent traffic over a VPN. Unfortunately, btpd does not currently allow you to bind to a specific IP address. 🙁
I decided to try to follow this guide.
Basically, you mark any packets and then do SNAT or DNAT or MASQUERADE to the packets, then use ip rules to force a specific route table.
In the end, my setup:
(To my router)
eth1 Link encap:Ethernet HWaddr 00:0a:cd:18:8a:ae
inet addr:172.29.5.10 Bcast:172.29.5.255 Mask:255.255.255.0
inet6 addr: fe80::20a:cdff:fe18:8aae/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2073338425 errors:0 dropped:0 overruns:0 frame:0
TX packets:2031270514 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3014149031 (2.8 GiB) TX bytes:1897259705 (1.7 GiB)
Interrupt:17 Base address:0x6c00
(my VPN)
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.1.1.4 P-t-P:10.1.1.4 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:7842859 errors:0 dropped:0 overruns:0 frame:0
TX packets:8040846 errors:0 dropped:41433 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:1544016294 (1.4 GiB) TX bytes:1737654278 (1.6 GiB)
iptables -t mangle -vnL (this is where I mark the packets):
Chain PREROUTING (policy ACCEPT 42 packets, 3595 bytes) pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 42 packets, 3595 bytes) pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 36 packets, 3461 bytes) pkts bytes target prot opt in out source destination
0 0 MARK icmp -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1000 MARK set 0x2a
0 0 MARK udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:!53 owner UID match 1000 MARK set 0x2a
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 owner UID match 1000 MARK set 0x2a
Chain POSTROUTING (policy ACCEPT 36 packets, 3461 bytes) pkts bytes target prot opt in out source destination
My iptables -t nat (Where I switch the source address):
Chain PREROUTING (policy ACCEPT 10 packets, 536 bytes) pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 10 packets, 536 bytes) pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 6 packets, 360 bytes) pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 6 packets, 360 bytes) pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * tun0 0.0.0.0/0 0.0.0.0/0
Output of ip rule show:
0: from all lookup local
32761: from all fwmark 0x2a lookup 42
32762: from 10.1.0.0/16 lookup 42
32766: from all lookup main
32767: from all lookup default
And finally, output of ip route show table 42:
default via 10.1.1.1 dev tun0
Now, everything seems to work fine when I do a tcpdump on my VPN interface at first. But, then it become apparent that my programs are not recieving the SYN-ACK of the threeway hand shake.
This is the output of the user with uid 1000 running curl ifconfig.me:
21:34:08.585901 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892988166 0,nop,wscale 6>
21:34:08.852160 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892988166>
21:34:11.587175 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892991168 0,nop,wscale 6>
21:34:11.848906 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892991168>
21:34:14.890209 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892991168>
21:34:17.603173 IP 10.1.1.4.55707 > www1465.sakura.ne.jp.http: S 342876994:3342876994(0) win 5840 <mss 1460,sackOK,timestamp 2892997184 0,nop,wscale 6>
21:34:17.863614 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:20.901869 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:26.962022 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
21:34:39.099655 IP www1465.sakura.ne.jp.http > 10.1.1.4.55707: S 530223170:3530223170(0) ack 3342876995 win 65535 <mss 1368,nop,wscale 6,sackOK,timestamp 248042701 2892997184>
As you can see, the masquerading seems to be working and I'm getting a return response. However, curl acts like it is not receiving the response. This is quickly verified with a netstat -an | grep SYN which shows curl was trying to use my internel network, not the VPN.
tcp 0 1 172.29.5.10:58000 219.94.163.75:80 SYN_SENT
I've tried doing SNAT and DNAT versus MASQUERADE to the same exact result. At this point, I'm not sure why curl is not recieving a response. Thanks.
Best Answer
Please verify that rp_filter is disabled. See section 10.1 of the LARTC HOWTO.. Also see this answer on a similar question
Also, if you do a
ip route show table main | grep 'dev tun0'you will probably see a route like below. You need to create a route like this yourtable 41.