Nginx – AWS WebServer architecture (ELB + iptables)

amazon-elbamazon-web-servicesiptablesnginxvarnish

I'm currently working on the architecture of high-traffic high-availability website.

We are using AWS.

Currently, we have:

Route 53 -> ELB -> Multiple EC2 instances -> RDS Multi AZ.

Each EC2 instance runs Varnish + Nginx & PHP FCGI. Sessions and some other shared data are stored via ElastiCache.

We plan to run Varnish and Nginx on each EC2 instance because this allow us to reduce failure points and to simply run extra instances if the load increases.

However, we have to add iptables (fail2ban) to this. Of course, we get the IP of the ELB instead of the real client ip…

We thought about the following solutions:

1) Add an EC2 instance between Route 53 and ELB, running iptables/firewalls (and maybe varnish ?) and forwarding everything to ELB.

2) Replace ELB by a custom EC2 running HAProxy + iptables.

3) Use mod_security and build some custom stuff to dynamically blacklist IPs

4) Stick with the current architecture and remove iptables from our todo-list.

What would be a good approach ?

Thanks

Best Answer

x-forwarded-for header

Of course, we get the IP of the ELB instead of the real client ip...

You'll want to make use of the x-forwarded-for header which will give you the client ip.

http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#x-forwarded-for

The X-Forwarded-For request header helps you identify the IP address of a client when you use HTTP/HTTPS load balancer.

Here's some other stuff you may be interested in ...

Security Groups & Network ACLs

AWS provide two great security features that operate in the same area iptables does:

Security Groups
Network ACLs

Security groups (think replacement for iptables) give you the ability to allow/deny traffic based on ip and port. There are other features around supplying another AWS resource in place of an ip address.

Network ACLs give you further control of your network. You can allow/deny traffic based on ip and port within and between subnets within your VPC.

AWS Security

If you're using VPC you have the ability to put things like EC2s in private subnets.
ELBs that need to be public facing can have just the essentials exposed (443, 80).
You can lock down services like SSH to a whitelist of IPs.
If you have an application with a web-interface for administration, you can lock it down at the web-server level to protect against any vulnerabilities at the software level.

Adding your own security to EC2s

If you're looking to add additional security measures on your EC2s (like fail2ban) and you're using multiple EC2s behind a load balancer, you'll need to engineer your EC2s so that they don't have any unique data on them (there's no point in adding a rule o EC2 #1 if it's not on EC2 #2).

There are techniques to keep EC2s in sync with each other, things like storing data on S3, pulling configuration management from a master, or pulling content from the same repo. Using Golden images along with bootstrap scripts (that pull data from S3).

Answers to your questions

1) Add an EC2 instance between Route 53 and ELB, running iptables/firewalls (and maybe varnish ?) and forwarding everything to ELB.

Adding an EC2 instance in front of your ELB is going to diminish your ability to scale. You also create a single point of failure so you likely do not want to do this.

2) Replace ELB by a custom EC2 running HAProxy + iptables.

You could do this but it comes at the cost of managing it yourself. You become responsibile for the software, security, updates, availability etc. You want to avoid this if possible (ie only do this if you can't solve the problem using an ELB).

3) Use mod_security and build some custom stuff to dynamically blacklist IPs

This is an option, but you may want to manage security at the OS/Network level, not (just) at the software/app level.

4) Stick with the current architecture and remove iptables from our todo-list.

Sure that's an option, but is that acceptable for the business/project?

Related Solutions

Nginx – Reverse Proxy Server Cloud Architecture (AWS + nginx)

I've set up this stack many times on AWS. Option #1 has always worked well for me and is the one I normally choose because it's the simplest. I'm curious to know how much traffic you're dealing with where the less-than-ideal initial cache hits are an issue? I'm serving a few million pageviews a month on a pair of m1.small instances and they're barely scratched.

Other thoughts:

Nginx can use memcached as a caching store. Plug it into an ElasticCache cluster. This will give you a very fast, centralized datastore for multiple instances to connect to for cache contents. (I've never done this, but it looks doable. Please post results if you do attempt it.)
The SPoF problem mentioned in Option #2 can be mitigated by setting up an autoscaling group that contains a single instance. If it dies, it should come back online in a matter of a couple of minutes. You will need to fiddle with a boot script that automatically grabs the Elastic IP though.
Keep in mind that putting anything in front of a ELB will require that you use VPC. This is generally good practice anyhow, but it can have a steep learning curve.
CloudWatch Alarm Actions were just announced today. You could do some cool stuff with this to help minimize the SPoF issue.

Nginx – ELB, nginx, and multiple HTTPS applications

You could run something like this:

          domain1.com  domain2.com  domain3.com
              \             |             /
               \            |            /
                \           |           /
                 \          |          /
                  \         |         /
                   \        |        /
                    \_______|_______/
                            |
ngix (static pages from local storage, 3 IP addresses, SSL termination)
                            |
                            |
                       ELB (optional)
                            |
                            |
                 -----------------------
                 |          |          |
                 V          V          V
                php        php        php

It would be a little bit cheaper, as 1 EC2 reserved small instance + 1 ELB is cheaper than 3 ELB.
You could be even cheaper if you could ditch ELB and just use normal DNS round robin (but you'll loose some ELB features like automatic failover).
Latency on serving static pages would not be affected by load on php-serving instances.
PHP serving would not compete for RAM with static pages OS cache.
You could even use apache mod_php instead of php-fpm is you don't need user separation between processes - it should be somewhat faster.
It would be cheaper to add more pages to serve (add one more IP instead of one more ELB).

But it also has some drawbacks:

You'd have have one more single point of failure (ngix server).
More complicated setup.