Linux – Huge performance penalty in gbit link with ip forwarding / masquerade through clear VPN tunnel

iptableslinuxnetworkingopenvpnperformance

I have a problem with IP forwarding between my home computer and a remote server (from online.net). My home connection is a 1 Gbit/s FTTH and the remote server has a 2.5 Gbit/s bandwidth.

The context

I'm running a clear OpenVPN tunnel between my home computer and a remote server at online.net. For the moment, I don't use any encryption so I don't face any performance penalty due to encryption.

Server machine : Intel Atom C2750 / 2GB of RAM / SSD
Client machine : VMware hosted Ubuntu server 16.04 64 bit with 8 GB of RAM / SSD / Intel i5 (Mac mini 2013 behind)

server openvpn command:

 # openvpn --dev tun --proto tcp-server --port 11000 --ifconfig 10.200.0.1 10.200.0.2 --cipher none --auth none --fragment 0 --mssfix 0 --tun-mtu 48000 --sndbuf 0 --rcvbuf 0

client openvpn command:

 # openvpn --dev tun --proto tcp-client --port 11000 --ifconfig 10.200.0.2 10.200.0.1 --cipher none --auth none --remote dedibox --fragment 0 --mssfix 0 --tun-mtu 48000 --sndbuf 0 --rcvbuf 0

I've set up a vsftpd server on the remote server. If I download something from the remote FTP server through the OpenVPN tunnel, I'm at full speed.

lftp aixki@10.200.0.1:~> pget 1GB.dat
1073741824 bytes transferred in 11 seconds (95.68 MiB/s)

The problem

The problem is when I want to access the internet from the client through the OpenVPN tunnel.

Let's add a route on the client machine for a particular website from which I test my download speed using wget/curl.

ip route add 62.34.91.0/24 via 10.200.0.1

Next, allow ipv4 forwarding on the remote server :

> # sysctl -w net.ipv4.ip_forward=1

And finally, allow VPN connections back and forth to the web :

iptables -t nat -A POSTROUTING -s 10.200.0.0/24 -o eth0 -j MASQUERADE

And when I test my download speed from the client :

wget -O /dev/null http://3-ipv4.testdebit.info/fichiers/1000Mo.dat
...
2016-05-17 20:24:45 (17.7 MB/s) - ‘/dev/null’ saved [1000000000/1000000000]

So that's almost 5 times less than when I download a file from the remote server using the VPN tunnel.

What could be the reason ?

  1. The link from the remote server to the 3-ipv4.testdebit.info website is slow ? No it's not, using the same wget from the remote server directly and I get 225 MB/s
  2. The remote server can't handle receiving and sending the same amount of data at the same time at 1 Gbit/s ? Yes it can. I've done an FTP file transfer from the server and downloaded at the same on the remote server using wget, I was at 100+ MB/s on both transfer.
  3. CPU overhead ? According to top, I'm using 15% of CPU capacity, so I don't think it's due to CPU.
  4. iptables masquerading all packets ? I think it is, but I'm not sure. Can someone confirm/deny ?

Any help is highly appreciated, it gives me headaches since two days ! If you need any logs or TCP dump, I can give it to you.

Edit 2016/05/22 at 14:09 GMT

I've dumped TCP packets using tcpdump on the server. All the dumps are available here :
http://163.172.210.224/dumps/

Here are the details about each file :

  • dump-server-direct-eth0.pcap Dump eth0 of a direct http download from the server. No VPN neither ip forwarding used here. Command used (I filter out my IP address used for ssh connection): tcpdump -i eth0 '((not net XXX.XXX.XXX.XXX) and port 80)' -C 100 -vvv -w dump-server-direct-eth0.pcap

  • dump-server-forward-http-[tcp|udp]-eth0.pcap Dump eth0 when the client download an http file through the UDP/TCP OpenVPN tunnel, using IP forwarding. Command used (I filter out my IP address used for ssh connection):
    tcpdump -i eth0 '((not net XXX.XXX.XXX.XXX) and (port 80 or port 11000))' -C 100 -vvv -w dump-server-forward-http-[tcp|udp]-eth0.pcap

  • dump-server-forward-http-[tcp|udp]-tun0.pcap Same as above, but this time capturing from tun0 interface. Command used:
    tcpdump -i tun0 -C 100 -vvv -w dump-server-forward-http-[tcp|udp]-tun0.pcap

  • dump-server-http-through-tcp-tunnel-eth0.pcap Dump eth0 when the client download an ftp file through the UDP/TCP OpenVPN tunnel, without IP forwarding. The http server is hosted on the OpenVPN server itself and I use the OpenVPN server address to connect to the server (10.200.0.1), so there's no IP forwarding at all. Command used (I filter with the client IP address):
    tcpdump -i eth0 'net YYY.YYY.YYY.YYY' -C 100 -vvv -w dump-server-ftp-through-tcp-tunnel eth0.pcap

  • dump-server-http-through-tcp-tunnel-tun0.pcap Same as above, but this time capturing from tun0 interface. Command used:
    tcpdump -i tun0 -C 100 -vvv -w dump-server-http-through-tcp-tunnel-tun0.pcap

Edit 2016/05/19 at 10:31 GMT

The comments and answers I get talk about the fact that I use TCP protocol with OpenVPN instead of udp. As I say in my post above, there's no speed performance issue between my server and my home computer through the TCP VPN tunnel :

lftp aixki@10.200.0.1:~> pget 1GB.dat
1073741824 bytes transferred in 11 seconds (95.68 MiB/s)

But to please everyone and stop the remarks about tcp versus udp, here's a test using udp :

server:

# openvpn --dev tun --proto udp --port 11000 --ifconfig 10.200.0.1 10.200.0.2 --cipher none --auth none --fragment 0 --mssfix 0 --tun-mtu 48000 --sndbuf 393216 --rcvbuf 393216

client:

# openvpn --dev tun --proto udp --port 11000 --ifconfig 10.200.0.2 10.200.0.1 --cipher none --auth none --fragment 0 --mssfix 0 --tun-mtu 48000 --sndbuf 393216 --rcvbuf 393216 --remote dedibox

iperf3 test from the server to the client using the udp vpn tunnel :

#  iperf3 -c 10.200.0.2  -i 1 -p 5201 -f m -b 1G -u

Connecting to host 10.200.0.2, port 5201
[  4] local 10.200.0.1 port 55287 connected to 10.200.0.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   111 MBytes   935 Mbits/sec  14265
[  4]   1.00-2.00   sec   119 MBytes  1000 Mbits/sec  15258
[  4]   2.00-3.00   sec   119 MBytes  1000 Mbits/sec  15258
[  4]   3.00-4.00   sec   119 MBytes  1000 Mbits/sec  15260
[  4]   4.00-5.00   sec   119 MBytes  1000 Mbits/sec  15259
[  4]   5.00-6.00   sec   119 MBytes  1000 Mbits/sec  15258
[  4]   6.00-7.00   sec   119 MBytes  1000 Mbits/sec  15253
[  4]   7.00-8.00   sec   119 MBytes  1000 Mbits/sec  15266
[  4]   8.00-9.00   sec   119 MBytes  1000 Mbits/sec  15255
[  4]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  15262
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.16 GBytes   993 Mbits/sec  0.763 ms  141473/150989 (94%)
[  4] Sent 150989 datagrams

iperf Done.

And finally, a speed test from the client and a speed test server hosted at testdebit.info, through my vpn tunnel :

# ip route add 62.34.91.3 via 10.200.0.1

# wget -O /dev/null http://3-ipv4.testdebit.info/fichiers/1000Mo.dat
/dev/null               100%[============================>] 953.67M  12.4MB/s    in 79s
2016-05-19 12:27:40 (12.1 MB/s) - ‘/dev/null’ saved [1000000000/1000000000]

Conclusion : no difference between UDP and TCP !

As I said, the problem is with iptables or ipv4.ip_forwarding, but not with OpenVPN / TCP / UDP

Best Answer

I have taken a very quick look at your last capture file. We can immediately see that the RTT on the end-to-end TCP session is around 50ms or worse. Your client is advaertising a window around 3MB (not 300MB as I originally typed). Put those together and you have an absolute maximum 480Mbps assuming infinitely fast hardware at each end of the TCP connection. Presumably neither end is infinitely fast.
So if I were you I would either:

accept the performance you get (depending upon actual requirements)

or:

graph the RTT across the whole session to gain accuracy
use bandwidth delay product to understand exactly what bandwidth is max for a single TCP flow.
investigate where the components of delay come from (e.g. by installing a tap at the tunnel server)

UPDATE: The delay is caused on the tunnel server. The capture files don't make this obvious but the end-to-end latency is much higher than I originally thought. The host-based capture files obscure the length of time each packet spends in the server. Ideally you should make a capture on the client and I think that will show much higher RTTs.

dump-server-forward-http-tcp-eth0 frame 17854 is a case in point. It contains data from the Internet host that is (supposedly) transmitted to the client in 17856. However 17856 is not acknowledged until 18614, some 222ms later. Allowing for a 40ms RTT to the client including some processing, then the server is adding at least 180ms to the one-way trip from reception from the Internet host to puttting it on the wire toward the client. Given that frame 17854 is clearly not a physical frame but has been reassembled (possibly erroneously) there may also be similar delays towards to Internet host.

If the RTT is 222 ms and the RWIN is 3MB then TCP can only deliver 13.5Mbps

It looks to me as though there is TCP segmentation offload enabled and that for whatever reason the TCP is being processed on the tunnel server when you really only want it forwarded. You now need to investigate why this might be the case.

UPDATE: I also just noticed that you say CPU is only 15%. If this refers to the tunnel server, you say that it's a C2750 Atom, 8-core. With a multi-thread CPU any sustained CPU load at or a bit above 100% / number-of-cores (in your case 12.5%) can be a single main thread saturating one core.

Related Topic