Blue/Green deployments with CloudFront

amazon-cloudfrontdeployment

I'm looking for a way to do Blue/Green deployments with CloudFront.

Does anyone have a good solution for moving from one CloudFront distribution to another or does everyone really just create their distribution and then never ever touch it again?

My CloudFront distribution consists of one S3 origin for static content (javascript, etc) and a custom origin pointing to an AWS ELB.

No Changes To CloudFront

Under normal circumstances we don't make any changes to our CloudFront distribution at all. We version our static content in the S3 origin by changing the name of the static content files in S3 and do rolling deployments to EC2 instances under the Elastic Load Balancer (ELB). However, there are times when we need to test and make changes to the CloudFront distribution itself or have significant enough changes to our environment that we need to point to a new ELB in a new environment.

Two CloudFront Distributions

The first option I attempted was to have two separate CloudFront Web Distributions, one for my current, or A, environment and one for my new, or B, environment. I attempted to use a Route53 weighted routing policy where I added two records for my www.domain.com Route53 record, one pointing to CloudFront Distribution A with a weight of 1 and the other pointing to CloudFront Distribution B with a weight of 0. The plan would be to change the weights when I want to move from distribution A to distribution B. However, only one CloudFront distribution at a time can have the www.domain.com Alternate Domain Names (CNAMEs) registered or you get the following error:

com.amazonaws.services.cloudfront.model.CNAMEAlreadyExistsException: One or more of the CNAMEs you provided are already associated with a different resource. (Service: AmazonCloudFront; Status Code: 409; Error Code: CNAMEAlreadyExists; Request ID: ef84a5f0-44e7-11e5-9315-0ba167bb108a)

One CloudFront Distribution

The second option is to keep one CloudFront web distribution.
I have S3 and custom origins pointing to both my A and B environments and then I update the CloudFront Cache Behavior to point to the other origin when I want to move from one environment to another. This is extremely messy because these updates take 15-60 minutes, there is no visibility into the progress of the update and depending on the nature of your change you may need to follow that up with a CloudFront Invalidation so you aren't serving cached content from the old environment along with new content.

Thanks for your advice!

Best Answer

Two Cloudfront Distributions

Since AWS allows for overlap between wildcard alternate CNAMEs in the same AWS account, you can switch between two cloudfront distributions in the following manner:

Use www.domain.com as Alternate CNAME for Prod distribution 1
Use *.domain.com as Alternate CNAME for Prod distribution 2
Point your CNAME DNS www.domain.com to distribution 1 or distribution 2. (*.cloudfront.net).
Remove the alternate CNAME from the distribution which you don't want to serve the content from anymore.

However, the two different distribution DNSs (*.cloudfront.net) may point to the same edge nodes, which means that the way your content is served is by matching the cloudfront.net CNAME to the Edge nodes serving it and then to match by hostname. In this case, if both your distributions are using the same edge nodes (it can be checked for example with dig) the cut will not be clean.

e.g. If both distributions share one or more edge nodes, distribution 1 with Alt CNAME www.domain.com will take precedence over distribution 2 with the more generic *.domain.com until the CNAME gets removed from the distribution 1 config in all edge nodes. So the version retrieved from one request may be different from the other during the transition period.

Related Solutions

CloudFront with Custom Origin and ELB

It's possible that CloudFront doesn't handle multiple headers with the same name correctly and isn't seeing your max-age directive. According to this CloudFront does use the Expires header if present, so try getting your origin server to set that instead (preferably relative to the request time). For Apache I think you want something like this with mod_expires:

ExpiresDefault "access plus 1 hour"

Blue/Green deployments with CloudFront with equal proportion

You can't do this with CloudFront.

tl;dr: your wildcard doesn't match hostnames where a specific, conflicting hostname is configured on another distribution.

You created the wildcard alternative hostname on distribution B as an attempt to work around this restriction:

You cannot add an alternate domain name to a CloudFront distribution if the alternate domain name already exists in another CloudFront distribution, even if your AWS account owns the other distribution.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CNAMEs.html#alternate-domain-names-restrictions

There is, of course, a reason for this restriction, and it also explains why distribution B would never see your requests, even though your DNS configuration is working as expected.

The exception to the rule...

However, you can add a wildcard alternate domain name, such as *.example.com, that includes (that overlaps with) a non-wildcard alternate domain name, such as www.example.com. Overlapping domain names can be in the same distribution or in separate distributions as long as both distributions were created by using the same AWS account.

... does not provide the exception you anticipated.

When a web browser connects to an endpoint, how the browser got there is not preserved -- was it a static A record, an Alias, a CNAME, a whole cascade of CNAMEs, or an entry in your hosts file? The server doesn't know, because that information is not preserved... It knows the IP address you arrived at, but that's from a pool shared by many distributions, so how your request happened to arrive at a particular CloudFront edge (which set of DNS records was followed, your "A" or "B" -- they may not even be different IP addresses on the CloudFront end) is not something that can be used to determine which distribution should service your request.

The only mechanism CloudFront has in order to determine which distribution should service a particular request is the HTTP Host: header in the incoming http request (potentially, SNI negotiation, too, but this doesn't change anything, whether or not CloudFront uses it).

Treating a request as belonging to a particular distribution is decided based on nothing else -- it can't be, since there's nothing else available to base it on.

By logical extension, only one distribution can be associated with any given incoming request Host: header, such as www.example.com (your distribution "A.")

The other distribution ("B"), *.example.com is, in fact, only able to serve requests for everything except www.example.com (or any other more specific alternate domain names you've associated with distributions that would otherwise match that wildcard) because another distribution on the same account with a more specific hostname associated ("A") claims the specific hostname www.example.com as an exception to the * wildcard.

Essentially, requests are checked for a distribution with a exact hostname match, first, and only when there is no match, would the distribution with the wildcard be used for the request.