I've recently configured a Debian 9 server (Debian 4.9.130-2) to run as a lightweight server, running a series of Docker containers (nextcloud, sync, etc.) alongside basic services like ssh. Services are properly configured and running without issue: I can connect to ssh and docker containers from any device on my LAN without any apparent troubles. However, attempts to connect from outside the network aren't getting to the server. At this point, I'm trying to test whether the failure is on the router or the server, as both seem to be properly configured. Towards this end, I set up packet capture on my router and then made several attempted inbound connection attempts to the forwarded ports from a VPN.
Here's a snippet from wireshark which indicates (I think) that traffic is indeed getting redirected by the router.
First, packet capture at the router:
No. Time Source Destination Protocol Length Info
2265 26.624915 196.52.84.12 87.75.107.144 TCP 80 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326616015 TSecr=0 SACK_PERM=1
2382 27.746737 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326617017 TSecr=0 SACK_PERM=1
2470 28.626743 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326618019 TSecr=0 SACK_PERM=1
2590 29.666995 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326619020 TSecr=0 SACK_PERM=1
2688 30.687513 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326620023 TSecr=0 SACK_PERM=1
2719 31.667451 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326621028 TSecr=0 SACK_PERM=1
2868 33.696000 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326623032 TSecr=0 SACK_PERM=1
3254 37.657240 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326627033 TSecr=0 SACK_PERM=1
3861 45.658800 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326635033 TSecr=0 SACK_PERM=1
4132 48.150464 196.52.84.12 87.75.107.144 TCP 80 57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326377039 TSecr=0 SACK_PERM=1
4152 49.191512 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326378040 TSecr=0 SACK_PERM=1
4207 50.160028 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57788 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326379041 TSecr=0 SACK_PERM=1
4464 52.415812 196.52.84.12 87.75.107.144 TCP 80 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326381262 TSecr=0 SACK_PERM=1
4530 53.412326 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326382263 TSecr=0 SACK_PERM=1
4631 54.373065 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326383263 TSecr=0 SACK_PERM=1
4684 55.380093 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326384264 TSecr=0 SACK_PERM=1
4779 56.420386 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326385265 TSecr=0 SACK_PERM=1
4874 57.420881 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326386265 TSecr=0 SACK_PERM=1
5161 59.374395 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 57789 → 2202 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326388265 TSecr=0 SACK_PERM=1
5381 61.774499 196.52.84.12 87.75.107.144 TCP 80 [TCP Retransmission] 54626 → 4003 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326651102 TSecr=0 SACK_PERM=1
Packet capture on server:
No. Time Source Destination Protocol Length Info
32179 24.444677474 196.52.84.12 192.168.1.208 TCP 78 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326616015 TSecr=0 SACK_PERM=1
33778 25.565718159 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326617017 TSecr=0 SACK_PERM=1
35147 26.445497552 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326618019 TSecr=0 SACK_PERM=1
36888 27.485382313 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326619020 TSecr=0 SACK_PERM=1
38683 28.505695805 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326620023 TSecr=0 SACK_PERM=1
40376 29.485394758 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326621028 TSecr=0 SACK_PERM=1
43649 31.513421847 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326623032 TSecr=0 SACK_PERM=1
50623 35.473792067 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326627033 TSecr=0 SACK_PERM=1
65139 43.473176096 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 54626 → 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326635033 TSecr=0 SACK_PERM=1
69018 45.964529458 196.52.84.12 192.168.1.208 TCP 78 57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326377039 TSecr=0 SACK_PERM=1
70816 47.004900826 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326378040 TSecr=0 SACK_PERM=1
72718 47.973061039 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57788 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326379041 TSecr=0 SACK_PERM=1
77788 50.228672533 196.52.84.12 192.168.1.208 TCP 78 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326381262 TSecr=0 SACK_PERM=1
80033 51.224501372 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326382263 TSecr=0 SACK_PERM=1
82529 52.185037535 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326383263 TSecr=0 SACK_PERM=1
84789 53.191738933 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326384264 TSecr=0 SACK_PERM=1
87000 54.231741538 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326385265 TSecr=0 SACK_PERM=1
88816 55.231936109 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326386265 TSecr=0 SACK_PERM=1
92836 57.184892614 196.52.84.12 192.168.1.208 TCP 78 [TCP Retransmission] 57789 → 22 [SYN] Seq=0 Win=65535 Len=0 MSS=1352 WS=64 TSval=326388265 TSecr=0 SACK_PERM=1
Quick key to output:
- 196.52.84.14 is an IP address assigned to my PC while connected to a VPN
- 192.168.1.208 is the LAN IP address for the server
- 87.75.107.144 is the WAN IP address on the router (obfuscated)
- port 2202 is being forwarded to port 22 for ssh on that server via tcp and port 4003 is being forwarded to 443 on the server
Am I right in thinking that the router is behaving as it should and forwarding the packets (e.g. "retransmission")?
Firewall is as follows:
$ sudo iptables-save
# Generated by iptables-save v1.6.0 on Fri Mar 15 20:37:38 2019
*nat
:PREROUTING ACCEPT [3920:488137]
:INPUT ACCEPT [2997:321060]
:OUTPUT ACCEPT [2725:243307]
:POSTROUTING ACCEPT [2735:246173]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-931904c155b2 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 8181 -j MASQUERADE
-A POSTROUTING -s 172.18.0.3/32 -d 172.18.0.3/32 -p tcp -m tcp --dport 7878 -j MASQUERADE
-A POSTROUTING -s 172.18.0.4/32 -d 172.18.0.4/32 -p tcp -m tcp --dport 8686 -j MASQUERADE
-A POSTROUTING -s 172.18.0.5/32 -d 172.18.0.5/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.18.0.6/32 -d 172.18.0.6/32 -p tcp -m tcp --dport 8989 -j MASQUERADE
-A POSTROUTING -s 172.18.0.7/32 -d 172.18.0.7/32 -p tcp -m tcp --dport 4040 -j MASQUERADE
-A POSTROUTING -s 172.18.0.8/32 -d 172.18.0.8/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 172.18.0.8/32 -d 172.18.0.8/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-931904c155b2 -j RETURN
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 8181 -j DNAT --to-destination 172.18.0.2:8181
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 7878 -j DNAT --to-destination 172.18.0.3:7878
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 8686 -j DNAT --to-destination 172.18.0.4:8686
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 9001 -j DNAT --to-destination 172.18.0.5:9000
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 27021 -j DNAT --to-destination 172.18.0.6:8989
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 4040 -j DNAT --to-destination 172.18.0.7:4040
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 10001 -j DNAT --to-destination 172.18.0.8:8000
-A DOCKER ! -i br-931904c155b2 -p tcp -m tcp --dport 10000 -j DNAT --to-destination 172.18.0.8:80
COMMIT
# Completed on Fri Mar 15 20:37:38 2019
# Generated by iptables-save v1.6.0 on Fri Mar 15 20:37:38 2019
*filter
:INPUT ACCEPT [6374971:555022347]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [8882591:15858115582]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m comment --comment "Allow SSH" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m comment --comment "Allow HTTPS" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-931904c155b2 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-931904c155b2 -j DOCKER
-A FORWARD -i br-931904c155b2 ! -o br-931904c155b2 -j ACCEPT
-A FORWARD -i br-931904c155b2 -o br-931904c155b2 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 7878 -j ACCEPT
-A DOCKER -d 172.18.0.4/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8686 -j ACCEPT
-A DOCKER -d 172.18.0.5/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 9000 -j ACCEPT
-A DOCKER -d 172.18.0.6/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8989 -j ACCEPT
-A DOCKER -d 172.18.0.7/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 4040 -j ACCEPT
-A DOCKER -d 172.18.0.8/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 8000 -j ACCEPT
-A DOCKER -d 172.18.0.8/32 ! -i br-931904c155b2 -o br-931904c155b2 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-931904c155b2 ! -o br-931904c155b2 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-931904c155b2 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Mar 15 20:37:38 2019
Routing table:
$ ip route
0.0.0.0/1 via 10.1.10.9 dev tun0
default via 192.168.1.1 dev eno1 onlink
10.1.10.1 via 10.1.10.9 dev tun0
10.1.10.9 dev tun0 proto kernel scope link src 10.1.10.10
128.0.0.0/1 via 10.1.10.9 dev tun0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-931904c155b2 proto kernel scope link src 172.18.0.1
172.98.67.82 via 192.168.1.1 dev eno1
192.0.0.0/8 dev eno1 proto kernel scope link src 192.168.1.208
192.168.1.0/24 via 192.168.1.1 dev eno1
Best Answer
The [TCP Retransmission] mean the packet was sent, and the PC still try to resend it, as it got no answer.
It usually mean the receiver didn't sent back a AC to aknowledge the data was received.
It could be bad route on the receiver's side, as often the receiver might not have a route back to your IP 196.52.84.14
I suggest debugging from the receiver's side, I suggest 192.168.1.208 as you can enable a packet sniffer there easily. Collect the log and check if the remote receiver know a default route to your VPN IP.
Edit 1
The wireshark show the machine get the retransmission but it don't answer. As such please make sure the firewall on the server does not block that traffic, winpcap/wireshark capture inbound packet BEFORE the local firewall discard those packets.