Ubuntu – How to block IPs that cause excessive 404 errors with Fail2ban

apache-2.4fail2banfirewalliptablesUbuntu

I have installed Fail2Ban v0.10.2 on Ubuntu 18.04 with Apache 2.4.29 and enabled the standard ssh and apache jails for basic protection with email notification warnings, when an IP is blocked.

Having a look at the documentation, I was not able to find a relevant filter that would help with the following situation:

I would like to ban IPs that hit the server and produce large numbers of 404 errors due to fake URL requests, which can be a typical spam bot behavior. So ideally, an IP is blocked that produces more than three 404 errors in a row with some exceptions for official search engine crawlers.

Is there a default regex for this situation?

I would appreciate your assistance on how to implement this.

Best Answer

I recommend you start by implementing the built-in apache-noscript filter for fail2ban. To do so, add the following lines to/etc/jail.local`

[apache-noscript] 
     enabled = true 
     port = http,https 
     filter = apache-noscript 
     logpath = /var/log/apache2/*error.log 
     maxretry = 3 
     bantime = 600

tweak the bantime setting to your liking and consider implementing the recidiv filter/jail for repeat offenders.

Note: there is a possible bug with the filter regex

Related Solutions

Fail2Ban – How to Unban an IP Properly

With Fail2Ban before v0.8.8:

fail2ban-client get YOURJAILNAMEHERE actionunban IPADDRESSHERE

With Fail2Ban v0.8.8 and later:

fail2ban-client set YOURJAILNAMEHERE unbanip IPADDRESSHERE

The hard part is finding the right jail:

Use iptables -L -n to find the rule name...
...then use fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g' to get the actual jail names. The rule name and jail name may not be the same but it should be clear which one is related to which.

Iptables – Why is iptables not blocking an IP address? (LB/proxy version)

Your iptables rules look fine. One can't tell for sure what they're passing, however, without logging accepts. Tcpdump won't tell you this because it runs on incoming before iptables runs. Since you have need of a load balancer, iptables-accept logs for port 80 would likely produce very large files that you'd need to manage carefully (for disk use) and you'd need scripts or other tools to analyze them. However, that's what you'd need to do to find out what's getting through.

What I can tell you from the above, though, is that there's an inherent leak problem in using network-packet string matching for application-level filtering. Packet boundaries do not respect application boundaries, so in a high-attack environment, some of these requests will leak through. This is a statistical effect. With enough requests directed at your system, the probability that some of them will be split into more than one packet increases. That alone can account for the leaks. Hackers can tilt the odds in their favor by inserting headers that increase packet size. But volume alone is enough.

For thorough filtering you need to filter at the application level. Apache provides several mechanisms for that. You can block users identified by fail2ban with a .htaccess rule for X-Forwarded-For. You can also filter in your Location directive. I don't see an option for doing this within fail2ban, nor a standalone utility that does this, but a custom script to sync fail2ban jail-ees with an apache filter would be one way to implement an application-level filter.

One more consideration: You posted rules for IPv4. If you're accepting IPv6 connections on port 80, you'll want to make sure those rules are also maintained.

Best Answer

Related Solutions

Fail2Ban – How to Unban an IP Properly

Iptables – Why is iptables not blocking an IP address? (LB/proxy version)

Related Topic