OK, so I have a VPC with three app servers and an instance of Postgres in RDS.
I have a security group called 'rds-staging' that allows inbound connections on port 5432 from a security group called 'app-elb-staging'.
'app-elb-staging' is the security group applied to all of my EC2 instances, and it allows outgoing traffic to go anywhere.
The RDS instance is in AZ us-east-1e. I can connect to it from my EC2 instance in us-east-1e (10.0.3.*), but not from any EC2 instances in us-east-1a (10.0.1.*) or us-east-1c (10.0.2.*):
deploy@ip-10-0-3-220:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
Connection to xxx.us-east-1.rds.amazonaws.com 5432 port [tcp/postgresql] succeeded!
deploy@ip-10-0-1-155:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
nc: connect to xxx.us-east-1.rds.amazonaws.com port 5432 (tcp) failed: No route to host
deploy@ip-10-0-2-90:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
nc: connect to xxx.us-east-1.rds.amazonaws.com port 5432 (tcp) failed: No route to host
Has anyone seen this before? I've checked the DNS, and each machine is resolving the hostname to the same IP (10.0.3.x).
Best Answer
OK, finally figured the root cause of this issue. The AMI I was using was creating a bridge that caused the connection issue due to it colliding with the IPs of my subnets. The output from
sudo route -n
looked like this on an affected instance:Any connection to 10.0.2.* would then fail:
Removing the bridge with
sudo ifconfig lxcbr0 down
resolved the issue, but using an AMI that does not set this bridge up in the first place corrected the root.