Using AWS WAF to Block Traffic to Lightsail Instance

I've set up this stack many times on AWS. Option #1 has always worked well for me and is the one I normally choose because it's the simplest. I'm curious to know how much traffic you're dealing with where the less-than-ideal initial cache hits are an issue? I'm serving a few million pageviews a month on a pair of m1.small instances and they're barely scratched.

Other thoughts:

• Nginx can use memcached as a caching store. Plug it into an ElasticCache cluster. This will give you a very fast, centralized datastore for multiple instances to connect to for cache contents. (I've never done this, but it looks doable. Please post results if you do attempt it.)

• The SPoF problem mentioned in Option #2 can be mitigated by setting up an autoscaling group that contains a single instance. If it dies, it should come back online in a matter of a couple of minutes. You will need to fiddle with a boot script that automatically grabs the Elastic IP though.

• Keep in mind that putting anything in front of a ELB will require that you use VPC. This is generally good practice anyhow, but it can have a steep learning curve.

• CloudWatch Alarm Actions were just announced today. You could do some cool stuff with this to help minimize the SPoF issue.

1) Setting Security Group is easy and important layer of security. If you are running any web app than port 80/443 are one that to be open to the world and 22 for accessing server remotely via ssh should be allowed from a particular IP address only. Other than these 3 ports all traffic will be blocked. You can test with Port Scanning or NMAP tool.

2) Limit access for SSH to a your static IP address only and also for accessing EC2 Instance via ssh you need a key. If attacker somehow get to know your IP address he/she cannot access your server without key.

Note :- If you are using CDN(CloudFlare) than your EC2 Static IP is already hidden.

3) You can limit the amount of concurrent connections from the same IP address to your server.

You can use linux firewall rules for that :-

iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP
iptables-save >/etc/iptables.up.rules

The first line will Watch the IP connecting to your eth0 interface. The second line will Check if the connection is new within the last 60 seconds and if the packet flow is higher than ten and if so it will drop the connection. The third line will Make the rules persistent in case of a reboot.

To verify the number of concurrent connections from all clients that are connected to your server :-

 netstat -tn 2>/dev/null | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head 

It will show a list of the current active connections by IP address and the offending IP is usually the one with a high number of connections.

12  10.1.1.1
160 162.19.17.93

In the example above the first number is the number of connections followed by the Originating IP address.

Note :- In a heavily loaded server the number of connections may be above 100, but during DDOS attack the number will go even higher. For an average host, if you have more than 30 connections from a single IP, chances are you are under attack. If more than 5 such IP/Host connected from same network , that's a very clear sign of DDOS attack.

Output of lsof,netstat and tcpdump are very useful in detecting such type of issues.

Now you get the IP address of the client you can use IPtables to block that IP or tcpkill command to do so. TCPKILL is part of dsniff package.

apt-get install dsniff

Then issue :-

tcpkill host x.x.x.x

The above method is good and it will help you to mitigate small DDOS Attack if applied correctly. Now if you are using CDN ( CloudFlare ) than you can block the attacker at that level only. You can use CloudFlare API to block the IP address. In this traffic will not come to you server.

Read more at CloudFlare API Doc

Refer to above method and create a script that will help you in automation.

4) In my opinion CloudFlare is better than CloudFront. CloudFlare is easy to setup and from one control panel you can handle everything. Even if you find heavy amount of unnecessary traffic than cloudflare "I am Under Attack" mode will mitigate it in under 5-10 seconds.

Read more about DDOS and I'm Under attack mode in Cloudflare Blogs.

5) You can setup AWS Alarms to stop/Terminate EC2 instance if your Network Bandwidth exceeds the limit.

AWS Alarm Sample

Edit:- One important thing is try to setup Monitoring tool (Like Nagios) and Log Management tool of web app access. This will help you to find the bottleneck.