How to temporarily redirect traffic from an Elastic Load Balancer

amazon-elbamazon-web-services

I'm running an application on Elastic Beanstalk, an whenever I need to deploy a new version, I need to:

  • deploy the version on Elastic Beanstalk, which takes a minute or so
  • patch the database with any SQL patches that come with the new version, which takes a few extra seconds

During that time, I'd like the ELB to stop redirecting traffic to the EC2 instances, and only bring back the traffic to them when it's safe to do so (when the software & database versions match).

Is it possible to temporarily redirect the traffic of the ELB somewhere else? Either performing an HTTP redirect to an S3 bucket for example, or reading from this bucket instead of an EC2 instance?

I know I could do this with DNS changes, but these would take extra time to propagate, and even with a low TTL, I still can't guarantee that clients will properly obey the TTL and reach the correct server as soon as I make the update.

So ideally, the DNS should always point to the ELB, and the ELB should then switch the traffic in real time to somewhere else just for the time of the upgrade.

Best Answer

AFAIK, there is no way to do it this way.

However, I found an alternative solution that checks all the boxes for me.

I updated my app's front controller to read an environment variable, APP_MAINTENANCE, and return a 503 Service Unavailable response if this variable is TRUE.

The database patching is run via container_commands and is automatic during deployment.

Now the deployment procedure is the following, using the Elastic Beanstalk API:

  • Use UpdateEnvironment to set APP_MAINTENANCE to TRUE
  • Wait until the environment is Ready (using DescribeEnvironments)
  • Use UpdateEnvironment to deploy the new application version
  • Wait until the environment is Ready
  • Use UpdateEnvironment to set APP_MAINTENANCE to FALSE.

This way, I can be sure that nothing is using the database during the database upgrade. During this timeframe (around 3 minutes), the application is unavailable and any request is returned a 503 response, which is acceptable in my use case.

Any client receiving this response is invited to try again in a few minutes, be it a real person (a friendly message will invite to do so), or an API consumer (HTTP suggests that this is a temporary failure, and that the client should retry later).

Related Topic