Resolve Public DNS Names via VPN in AWS

f you know any thing about this (maybe you are a network engineer) you probably see right away what the problem is.
The problem is rooted in these concepts:

• Ephemeral ports
• Stateless and Stateful
• Connection Tracking
• NACLs are Stateless
• Security Groups are Stateful

Read about it here:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html

https://en.wikipedia.org/wiki/Ephemeral_port

The corrected confiuration is below. The change that really corrected the connectivity between the Public and Private subnets was enabling response traffic on the Ephemeral ports for both In Bound Public subnet NACL and Out Bound Private subnet NACL rules. I also cleaned up some redundant and insecure Out Bound rules in the NACLS and Security Groups.

VPC NACL

In Bound
80  0.0.0.0/0 Allow
Out Bound
32768-65535 0.0.0.0/0 Allow

Public Subnet NACL

InBound
80  0.0.0.0/0 Allow
32768-65535 172.30.4.0/0 Allow
Out Bound
32768-65535 0.0.0.0/0 Allow
5432    172.30.4.0/24 Allow (PostgreSQL)

Private Subnet NACL

InBound
5432    172.30.1.0/24 Allow
Out Bound
32768-65535 172.30.1.0/24 Allow 

VPC Security Group

InBound
80  VPC-Security-Group-ID Allow
Out Bound

Public Subnet Security Group

InBound
80  0.0.0.0/0 Allow
Out Bound
5432    172.30.4.0/0 Allow

Private Subnet Security Group

InBound
5432    172.30.1.0/24 Allow
Out Bound

This is a working configuration. Hopefully this helps someone. If you see anyhting that can be improved let me know and let me know if you have questions.

I had opened a ticket in AWS Support and they confirmed that is working as designed: traffic will ONLY work in one direction. It is because client's IP is NAT'd. Here is AWS Support response for my ticket response:

I understand that you have a client VPN (CVPN) set up and you were able to successfully connect (initiate TCP connection) to EC2 instances, but the EC2 instances cannot connect to the client IP assigned to the clients.

As stated in the following link: https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-aws-client-vpn-to-securely-access-aws-and-on-premises-resources/

Note that CVPN will use source NAT (SNAT) to connect to resources in the associated VPC(s).

So, any traffic initiated from the client's IP will be NAT'd (source IP of the client will be changed) to the IP address of the CVPN endpoint. So, the EC2 instance will see as if the traffic is being sourced from the CVPN endpoint IP and not from the client's IP. Also, the VPC route table will not have a route to the client IP's subnet (refer to VPC route table rtb-0ef010cd7b387b8ff). Hence, the connection can only be initiated from the client to the EC2 instance and it wouldn't work for the connections initiated in the other direction.