Openvpn – Connecting an EC2 VPC with OpenVPN all routed traffic being lost

amazon ec2amazon-vpcopenvpn

I'm trying to use OpenVPN on Amazon Linux to connect the local LAN to a VPC on EC2. I have traffic flowing from the OpenVPN instance to any machine on the LAN, but other machines on the VPC are not seeing traffic from the LAN.

Here's a trimmed version of the network:

       Local network          /        EC2 VPC, 10.2.0.*/255.255.255.0
  10.1.0.*/               ,'
     255.255.255.0     ,'
                      .'
                      |
 +---------------+   |             
 | OpenVPN on    |   |             
 |   firewall    XXX |   +---------------+
 | 10.1.0.1      |  XXXX | OpenVPN server|
 +------`.-------+   \  XX  10.2.0.10  `-.    +-------------+
          \          '.  +---------------+  `-. Second server
  +--------`.---+     |                       | 10.2.0.12   |
  |Local server |      \                      +-------------+
  | 10.1.0.3    |       \
  +-------------+        \

From the EC2 OpenVPN server:

[root@ip-10-2-0-10 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.2.0.1        0.0.0.0         UG    0      0        0 eth0
10.1.0.0        10.1.2.5        255.255.255.0   UG    0      0        0 tun0
10.1.2.0        10.1.2.5        255.255.255.0   UG    0      0        0 tun0
10.1.2.5        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.2.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

[root@ip-10-2-0-10 ~]# ping -c 2 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.
64 bytes from 10.1.0.3: icmp_seq=1 ttl=127 time=488 ms
64 bytes from 10.1.0.3: icmp_seq=2 ttl=127 time=54.9 ms

--- 10.1.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1056ms
rtt min/avg/max/mdev = 54.925/271.767/488.609/216.842 ms

From the LAN firewall:

root@firewall:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         xxx.xxx.xxx.xxx 0.0.0.0         UG    0      0        0 eth0.2
10.1.0.0        0.0.0.0         255.255.255.0   U     0      0        0 br-lan
10.1.2.0        10.1.2.2        255.255.255.0   UG    0      0        0 tun0
10.1.2.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.2.0.0        10.1.2.2        255.255.255.0   UG    0      0        0 tun0

From the Second server on EC2:

[root@ip-10-2-0-12 ~]# ping -c 3 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.

--- 10.1.0.3 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 12014ms

However here's the kicker. Doing a tcpdump on the EC2 OpenVPN server shows all traffic flowing as it should:

[root@ip-10-2-0-10 ~]# tcpdump -i eth0 -n host 10.1.0.3
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:58.779826 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 1, length 64
13:46:58.852756 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 1, length 64
13:46:59.787467 IP 10.2.0.12 > 10.1.0.3: ICMP echo request, id 21846, seq 2, length 64
13:46:59.847424 IP 10.1.0.3 > 10.2.0.12: ICMP echo reply, id 21846, seq 2, length 64

Tcpdump on the Second server on EC2 only shows the echo request and no replies.

  • IPtables firewalls are disabled on all EC2 instances.
  • IP forwarding has been enabled on the OpenVPN server.
  • The EC2 security group that both the OpenVPN server and Second server are set to allow all traffic from 10.0.0.0/8 and the network ACLs are wide open.

It appears this traffic is being dropped by Amazon that originates outside the VPC, regardless of the Security Group settings.

Anything I am missing that would be causing this traffic to be dropped?

Best Answer

Turns out EC2 was blocking the forwarded packets from the OpenVPN server.

There is a setting on the EC2 Dashboard under Network & Security -> Network Interfaces -> Actions -> Change Source/Dest. Check.

When I disabled this on all my instances traffic through the VPN is flowing as intended.

Hopefully this will help someone else.