Nat – AWS private subject cannot hit S3 using VPC endpoint

Publicly-routable IP addresses don't get rewritten when they pass through the NAT instance.

You'll need to leave the entire public Internet address space as permissible on the private subnet in the network ACLs. If the private subnet lacks an Internet gateway and its default route points to the NAT instance, public Internet addresses will only arrive indirectly via the NAT instance.

VPC network ACLs are useful to limit access between instances inside a VPC, but their stateless nature makes them cumbersome for the type of configuration you describe: it doesn't keep track of a connection that matched an allowed outbound rule to permit the corresponding inbound traffic, so you're forced to approximate by allowing ephemeral port ranges for inbound traffic.

A more flexible approach is to use a combination of VPC routing, the absence of an Internet gateway on the private subnet, and a good iptables configuration in the NAT instance to control traffic to and from publicly routable IP space, while leaving the network ACL for private subnet instances permissive-by-default with respect to publicly routable IP space. In such an environment, placement in the private subnet is sufficient to protect instances from any outside traffic the NAT instance does not pass.

The problem:

standard AMI instance not working correctly as a NAT instance

If you try to use a standard AMI instance as a NAT instance it'll have the same symptoms as described in the question above:

• SSH connection to the NAT = OK
• SSH connection to private subnet instance = OK
• ping to/from private subnet instance = OK
• private subnet instance outbound calls = FAIL (even with valid route tables/security groups)

The hard solution:

To get a standard AMI instance working as a NAT instance you need to customise it:

• modify the iptables as shown here
• ensure IPv4 forwarding is enabled and ICMP redirects are disabled as noted here

The easy solution:

use a Community amzn-ami-vpc-nat instance already configured for use as a NAT instance

For most people (who are using the NAT instance for admin connection purposes) a customised NAT instance is simply unnecessary.

AWS provides standard instances configured for NAT use (ie modified as described above) as community AMI instances.

• click Launch Instance in the EC2 control panel
• in Step 1: Choose an Amazon Machine Image (AMI) click the Community AMIs selector on the left
• type amzn-ami-vpc-nat 2017 or more simply nat 2017 (note below) in the search input box
• launch your NAT and ensure your route tables + security groups are correct - see here

Note:
The reason you include the year in the Community AMI instance search is that AWS keeps all the old NAT AMI versions (21 at the time of writing, going back to 2013) - which mostly include a year in the version number
...it's best / easiest to simply choose the latest version.