ELB – High latency when enabling cross-zone balancing

amazon ec2amazon-elbamazon-web-serviceslatency

I'm working on improve our infrastructure reliability in AWS and one of my tasks is enabling cross-zone balancing on our ELBs.
Our ELBs receives around 20K request by second on rush hours as well as all our instances are in us-east-1d.
I've followed the below steps:

  1. Modified the ASG to start creating instances in us-east-1c and duplicating the desired capacity
  2. Wait until the new instances created be fully provisioned
  3. Enable us-east-1c zone in ELB.

When I perform the step 3 receive a spike of Sum ELB 5XXs errors.

I guess that these errors are related to that our recently escalated servers are "cold". There are a way to start sending request to them incrementally?

Thanks.

Best Answer

When it comes to pre-warming your load balancer, AWS does not come with any built-in feature for that. They suggest that you do your own "load test" script that increase traffic incrementally OR to contact them and they will have it pre-warmed for you.

When you add another availability zone that your load balancer should include, I believe that the whole load balancer is replaced with a new one behind the scenes (at least that's what happens when you change the AvailabilityZones property through CloudFormation). That means that the newly created load balancer is not warmed up and will need a couple of minutes to scale up.

My suggestion is that you:

  1. Setup a completely new load balancer
  2. Configure it as your old one but with the new availability zone added
  3. Ask Amazon to pre-warm it for you
  4. Make the DNS switch when it's as "hot" as possible
Related Topic