WARNING: Long. Lots of info here.
3 years ago someone asked Why is iptables not blocking an IP address? and it turned out the reason was because the servers were behind CloudFlare which made it impossible to block IP addresses directly they way they wanted to unless you use it differently. Any reverse proxy or load balancer would cause the same thing.
Similarly we have setup fail2ban with a rule to ban any bots which attempt to brute-force their way into the administrative login or spam xmlrpc. The site is sitting behind a load balancer so obviously we can't directly ban the IP address but iptables is supposed to be accepting the connection and pattern matching the packet data to ban specific traffic.
This is fail2ban jail.conf config:
[wp-auth]
enabled = true
filter = wp-auth
action = iptables-proxy[name = lb, port = http, protocol = tcp]
sendmail-whois[name=LoginDetect, dest=ITemail@ourdomain.com, sender=acceptablebotbot@ourdomain.com, sendername="Fail2Ban"]
logpath = /obfuscated/path/to/site/transfer_log
bantime = 604800
maxretry = 4
findtime = 120
This is the simply pattern match for wp-login requests:
[Definition]
failregex = ^<HOST> .* "POST /wp-login.php
ignoreip = # our ip address
This is our fail2ban iptables action which is supposed to be able to block these bots but for the most part doesn't seem to. It is from the CentOS site Tips section for fail2ban behind a proxy. For the sake of brevity I've left only the section header comments in place.
# Fail2Ban configuration file
#
# Author: Centos.Tips
#
[INCLUDES]
before = iptables-blocktype.conf
[Definition]
# Option: actionstart
actionstart = iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>
# Option: actionstop
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
iptables -F fail2ban-<name>
iptables -X fail2ban-<name>
# Option: actioncheck
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'
# Option: actionban
actionban = iptables -I fail2ban-<name> 1 -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP
# Option: actionunban
actionunban = iptables -D fail2ban-<name> -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP
[Init]
# Default name of the chain
name = default
# Option: port
port = http
# Option: protocol
protocol = tcp
# Option: chain
chain = INPUT
So as I mentioned the site is on a pair of servers behind an elastic load balancer and seems to work in test. We can add any of our own IP addresses and we cannot reach the site. Despite this bots seem to be able to get through.
[root:~/] iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-SSH
-N fail2ban-lb
-A INPUT -p tcp -m tcp --dport 80 -j fail2ban-lb
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 24007:24020 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A fail2ban-SSH -j RETURN
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.200.12.33" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.134.50.10" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 160.202.163.125" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 162.243.68.232" --algo bm --to 65535 -j DROP
-A fail2ban-lb -j RETURN
Port 80 is the only port open to all. All others are ACL'd via AWS Security Groups. IPtables appears to be processing in the correct order and should therefore be blocking these IPs based on their X-Forwarded-For header. There is a Firefox plugin which allows you to send these headers with initial requests and we get blocked as a result with any of these bot IPs as well.
The source IP address does not appear to be forging the X-Forwarded-For header as we've been playing with as the ELB rewrites them anyway. tcpdump does not show any extra information on the packet at the server level.
22:07:14.309998 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 2545:3054, ack 19506, win 166, options [nop,nop,TS val 592575835 ecr 2772410449], length 509
E..1..@.@..9
...
f.p+..P.Nz.
20............
#Q.[.?.QPOST /wp-login.php HTTP/1.1
host: www.thiswebsite.com
Accept: */*
Accept-Language: zh-cn
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Referer: http://www.thiswebsite.com/wp-login.php
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)
X-Forwarded-For: 91.200.12.33
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 21
Connection: keep-alive
These requests are all being logged in the transfer_log.
When we do the same thing and forge the X-Forwarded-For we get caught by iptables before ever reaching Apache. tcpdump also shows our extra IPs.
20:10:25.378873 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 3157:3860, ack 124583, win 267, options [nop,nop,TS val 526293643 ecr 2507283790], length 703
E...Tf@.@.[.
...
f.p,O.P...GU........m.....
.^...r.QPOST /wp-login.php HTTP/1.1
host: www.thiswebsite.com
Accept: /
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Cookie: __utma=190528439.16251225.1476378792.1478280188.1478289736.3; __utmz=190528439.1476378792.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _icl_current_language=en; __utmc=190528439; __utmb=190528439.2.10.1478289736; __utmt=1
Pragma: no-cache
Referer: http://www.thiswebsite.com/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0
X-Forwarded-For: 91.200.12.33, <our ip address>
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Connection: keep-alive
I also have the ELB access log here which I expect to see an entry for, just not the Apache transfer logs.
2016-11-07T22:07:14.309917Z mLB 91.200.12.33:60407 10.4.8.71:80 0.000079 1.99244 0.000091 200 200 21 3245 "POST http://www.thiswebsite.com:80/wp-login.php HTTP/1.1" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)" - -
So the IP address (at least according to the ELB) does not appear to be forced at the X-Forwarded-For level. Why is traffic from it not being blocked? The IP address also shows up constantly in the fail2ban log with the usual:
fail2ban.actions[11535]: INFO [wp-auth] 91.200.12.33 already banned
Best Answer
Your iptables rules look fine. One can't tell for sure what they're passing, however, without logging accepts. Tcpdump won't tell you this because it runs on incoming before iptables runs. Since you have need of a load balancer, iptables-accept logs for port 80 would likely produce very large files that you'd need to manage carefully (for disk use) and you'd need scripts or other tools to analyze them. However, that's what you'd need to do to find out what's getting through.
What I can tell you from the above, though, is that there's an inherent leak problem in using network-packet string matching for application-level filtering. Packet boundaries do not respect application boundaries, so in a high-attack environment, some of these requests will leak through. This is a statistical effect. With enough requests directed at your system, the probability that some of them will be split into more than one packet increases. That alone can account for the leaks. Hackers can tilt the odds in their favor by inserting headers that increase packet size. But volume alone is enough.
For thorough filtering you need to filter at the application level. Apache provides several mechanisms for that. You can block users identified by fail2ban with a .htaccess rule for X-Forwarded-For. You can also filter in your Location directive. I don't see an option for doing this within fail2ban, nor a standalone utility that does this, but a custom script to sync fail2ban jail-ees with an apache filter would be one way to implement an application-level filter.
One more consideration: You posted rules for IPv4. If you're accepting IPv6 connections on port 80, you'll want to make sure those rules are also maintained.