Nat – AWS load balancer, multiple instances – NAT

Sounds like requests queuing in your ELBs due to (maybe) long Keep-Alive headers. ELBs will try to minimize the number of connections to the backend by honoring the Keep-Alive header.

Short (or disabled) keep-alive will force ELB to open a single connection per request to your backend, should your backend nodes have free resources to handle that incoming connection. Long keep-alives will reduce backend connections and will enforce the ELBs to serialize requests as much as possible, which may result in requests waiting for others being processed.

Check out your ELB queues in Cloudwatch during load tests and, if they grow while your backends are idle, try lowering the number of requests per connection and reducing the Keep-Alive timeouts. The exact values will vary depending of your application requirements.

PS: t1.micro is evil PS(2): Stickyness is evil

At its heart, an ELB is just a collection of EC2 instances. When you create an ELB, you specify the availability zones you want that load balancer to be in. Instances to make up the load balancer will then be created in those zones. The way they avoid a single point of failure here is by returning multiple IP addresses when you do a DNS lookup. For example:

• DNS lookup for website.example.com returns CNAME website-elb-12345.eu-west-1.elb.amazonaws.com
• The lookup also returns the information for website-elb-12345.eu-west-1.elb.amazonaws.com. It states that the site has IP address 1.2.3.4 and IP address 2.3.4.5

It is up to the client to choose which IP address to use to make a connection. The IP addresses won't always be returned in the same order from the DNS lookup. A client could retry on an alternative IP address if they can't connect on the first attempt.

The TTL on the DNS records for an ELB is only 60 seconds which means that should an ELB instance die and get replaced, the DNS will be updated everywhere fairly quickly.