I've got an API app on EC2 that only knows HTTP, and it's listening on port 8080. Its code does not return 504 for any request. I need CloudFront to be an SSL-Terminating reverse proxy because the app only speaks HTTP.
My API is behind CloudFront. When I visit my custom domain, the static web pages are being returned from S3 just fine, but the traffic for the API 504s (The request could not be satisfied) or sometimes times out.
TL/DR
CloudFront is returning 504 (The request could not be satisfied) to users requests to an API (on EC2 behind CloudFront). Requests to S3 work properly.
Security Groups
My Security Groups for EC2 are configured to be auto updated by a Lambda with the latest CF IPs (https://aws.amazon.com/blogs/security/how-to-automatically-update-your-security-groups-for-amazon-cloudfront-and-aws-waf-by-using-aws-lambda/). I tried having the HTTP and HTTPS ingress rules open up ports 8080 and 80. Neither solve the issue.
In CloudFront, the Alternate Domain Names (CNAMEs) has my custom domain listed. I have an SSL Certificate (ACM issued) and it looks like its working fine (S3 assets are being returned normally in the browser).
CloudFront Distribution Settings
CloudFront Origin Settings
CloudFront Behavior Settings (precedent 0)
CloudFront Behaviors
Tried
-
If I make an ingress for the whole world in the security group, and curl the API from home, I do get a valid reply, but CloudFront proxied requests still 504.
-
Making the HTTP and HTTPS ingress rules open up ports 8080 and 80. Neither solve the issue
-
How do I resolve the error "The request could not be satisfied. Bad Request" from Amazon CloudFront?
I'm assuming that my traffic from CloudFront is not reaching my API applicaiton. Anyone know what I need to do to fix this? This is my first time setting CloudFront up as an SSL-terminating reverse proxy.
Best Answer
Finally found out the source of the issue after a few days of pulling levers.
CloudFront -> Distributions -> Distribution Settings -> Behaviors Tab -> Behavior with the Path Pattern
/api/*I needed to change the Cache Based on Selected Request Headers setting to All.
Initially (shown in the screenshot in the question) I had it set to None (Improves Caching). If I toggle this single setting back to that, the exact same issue returns after about a minute, so this appears to be the crux.
With this newly changed setting, it means that CloudFront does no caching for requests to the API. So it's a good practice to implement a WAF for rate limiting, IP blocking, geo blocking. Also it's a good practice to put a load balancer in front of the API application so it can distribute load to multiple app instances. And maybe even application-level caching depending on your use-case.