Elastic beanstalk deployment in private subnet of the VPC is failing with following error when public subnet ACL inbound is deny all

amazon-beanstalkamazon-vpcelastic-beanstalknetworkingsubnet

TLD: Error, when the public subnet is inbound, is deny all.

The EC2 instances failed to communicate with AWS Elastic Beanstalk,  
either because of configuration problems with the VPC or a failed EC2 instance.  
Check your VPC configuration and try launching the environment again.

my configuration

I have three subnets inside a VPC also public is assigned with internet age way and both private subnets are assigned with nat gateway
enter image description here
VPC endpoint services added with SG allowing all traffic from VPC level and added both private-1 and private-2 subnets
enter image description here

public subnet ACL inbound
enter image description here

public subnet ACL outbound
enter image description here

private-1 and private-2 subnets ACL inbound
enter image description here

private-1 and private-2 subnets ACL outbound
enter image description here

Using a cloud formation template deployed a node beanstalk app both ec2 and ELB in the private-1 subnet where ELB is with the internal schema security group allows port 80 on instance from ELB SG.

And after a long wait, the beanstalk app failed with the error 

The EC2 instances failed to communicate with AWS Elastic Beanstalk,  
either because of configuration problems with the VPC or a failed EC2 instance.  
Check your VPC configuration and try launching the environment again.

I allowed ports from public subnet to private-1 and connected through ssh tunnel from instance public subnet into the ec2 instance created by the beanstalk.
There are is no configuration done on the instance

  1. Node app is not running on port 8081
  2. Nginx is not running on port 8080
  3. No IP tables forward rule for port 80 to 8080
    enter image description here

When I allow inbound on public subnet elastic beanstalk app is configured and the environment is green
enter image description here

I don't understand why I need to allow public subnet ACL inbound if I'm not at all using any resource in that and why beanstalk is not configuring the instance in that case?

I still face the issue after allowing 1024-65535 ephemeral ports in public subnet inbound ACL for return traffic from the internet.

Beanstalk able to launch successfully after adding port 443 in public subnet inbound ACL
new ACL for the public subnet
enter image description here

As no network allowed from public subnet to private subnet and the entire setup is in private subnet, why port 443 breaking the beanstalk environment?

Best Answer

I really don't understand why I need to allow public subnet ACL inbound if I'm not at all using any resource in that

But you are. You have a NAT Gateway. NAT Gateways provide service for resources on private subnets, but they are actually located on public subnets, and use the public subnet on their Internet-facing side.

You are blocking reply traffic from the Internet from returning to the NAT Gateway.

and why beanstalk is not configuring the instance in that case?

The reason for this should be apparent from the above. As the error message indicates, the instance didn't contact the EB service, because it can't -- you blocked it -- so no provisioning can happen.

You can use any VPC configuration that you like as long as it meets the following requirements.

VPC Requirements

Internet Access – Instances must have access to the Internet through one of the following methods.

  • Public Subnet – Instances have a public IP address and use an Internet Gateway to access the Internet.

  • Private Subnet – Instances use a NAT device to access the Internet.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/vpc.html

Network ACLs are best left alone unless you have a specific reason to configure them. Security Groups are sufficient for most purposes.

Related Topic