A client's website is currently under attack, and I've been called in to fix it.
A huge number of IPs (easily over 5,000) is constantly hitting /login
, presumably trying to bruteforce their way in.
I've changed the site so the page returns a 500 error, but they're not giving up. Obviously this is no good for the real users, who now can't log in.
Load balancing is done via HAProxy, which I have very little knowledge of (although I have a lot more than I did a few hours ago). I've tried lots of sensible things that I've found online but none seem to have helped, probably because there is such a big block of IPs performing the attack.
Onto the question, then:
How can I reject an IP if it hits /login
more than X times in Y seconds?
And, subpoint – how can I see a log of rejections so I know it's actually working?
Here is a sample of the haproxy.log:
Jun 3 14:24:50 hap-server haproxy[11831]: 46.161.62.79:15290 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-03 751/0/202/38/991 500 220 - - ---- 428/428/120/38/0 0/0 "GET /login HTTP/1.1"
Jun 3 14:24:50 hap-server haproxy[11831]: 46.161.63.132:47804 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-04 751/0/202/38/991 500 220 - - ---- 428/428/119/42/0 0/0 "GET /login HTTP/1.1"
Jun 3 14:24:50 hap-server haproxy[11831]: 46.161.62.43:53176 [03/Jun/2017:14:24:49.505] www-https-test~ www-backend/www-02 751/0/202/38/991 500 220 - - ---- 428/428/118/38/0 0/0 "GET /login HTTP/1.1"
Best Answer
Thanks to a comment by EEAA, I was able to solve this using fail2ban.
There's very little documentation about how to use fail2ban with HAProxy, however - so little in fact that this page is already nearing the top of a Google search for "haproxy fail2ban", so I'll detail how I did it.
First of all, install fail2ban. If you can't do that bit you probably shouldn't carry on without some more help.
fail2ban works by scanning your access logs, looking for a pattern you set up. If it finds that pattern X times in Y seconds, it will automatically ban that IP for Z seconds.
Your HAProxy log should be at
/var/log/haproxy.log
. If you're under heavy load it might be too big to open with Vim or Nano, so just look at the last few lines by using tail:tail -n50 /var/log/haproxy.log
A sample of a "bad" entry in my log looked like this:
The important bits we want fail2ban to get hold of are the IP of the attacker and the page they are hitting.
To tell fail2ban how to do this, first we want to create a filter. Create a file in the folder
/etc/fail2ban/filter.d
. I called mineapplogin.conf
but you can call it what you like, as long as it ends in.conf
.The contents were as follows:
<HOST>
is the point in the line from your log where the IP address appears. If you want to use the same regex as me, replace/login
with the address the attackers are targeting on your server.Now you have to tell fail2ban that you want it to look for that filter. Make a copy of
/etc/fail2ban/jail.conf
(in case you mess up and need to start again), then openjail.conf
and add the following lines at the bottom.The part in the square brackets is just a name - use what you like.
bantime
is the number of seconds a user is banned for.maxretry
is the amount of times a user can hit the page infindtime
seconds. So, in my example, if someone tries to get to www.mydomain.com/login more than six times (or maybe exactly six times, I'm not sure) in 2 minutes then they will be banned for 20 minutes.filter
is the name of the file you created in/etc/fail2ban/filter.d
(but without the.conf
).logpath
is the path to the log file you want it to search in.Save that file, then restart fail2ban:
service fail2ban restart
Tail the fail2ban logfile:tail -f /var/log/fail2ban.log
and hopefully, after a minute or two, you'll start to see something likeThat's it! You're safe from baddies (the ones that try to get to that specific page, anyway).
Note - you might also see some SSH bans going on in your fail2ban log. It creates some rules for SSH automatically when it's installed, so don't panic.