I have configured fail2ban to monitor a certain pattern of malicious traffic I'm getting and ban IP addresses associated.
Everything seems to be working great — the regex is matching the pattern appropriately and the problem IP address is getting added to iptables.
However, when I check the Apache logs I'm still getting hits from the IP address that's getting banned. It is as though iptables is not running at all.
So let me share some specifics just to confirm that everything is configured correctly.
First, I'll clear and reload the iptables rules:
$ sudo iptables -F
$ cat /etc/iptables.firewall.rules
*filter
# Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT
# Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT
# Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
# Allow SSH connections
#
# The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Allow ping
-A INPUT -p icmp -j ACCEPT
# Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
# Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP
COMMIT
$ sudo iptables-restore < /etc/iptables.firewall.rules
$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * * 0.0.0.0/0 127.0.0.0/8 reject-with icmp-port-unreachable
14 1432 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 5/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
11 1638 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Now, here's what the fail2ban configuration looks like:
$ cat /etc/fail2ban/filter.d/apache-xmlrpc.conf
[Definition]
failregex = .*:80 <HOST> .*POST .*xmlrpc\.php.*
ignoreregex =
$ cat /etc/fail2ban/jail.local
[apache-xmlrpc]
enabled = true
port = http,https
filter = apache-xmlrpc
logpath = /var/log/apache2/other_vhosts_access.log
maxretry = 6
$ fail2ban-regex /var/log/apache2/other_vhosts_access.log /etc/fail2ban/filter.d/apache-xmlrpc.conf
Running tests
=============
Use regex file : /etc/fail2ban/filter.d/apache-xmlrpc.conf
Use log file : /var/log/apache2/other_vhosts_access.log
Results
=======
Failregex
|- Regular expressions:
| [1] .*:80 <HOST> .*POST .*xmlrpc\.php.*
|
`- Number of matches:
[1] 29 match(es)
Ignoreregex
|- Regular expressions:
|
`- Number of matches:
Summary
=======
Addresses found:
[1]
80.82.70.239 (Sat Jul 13 02:41:52 2013)
80.82.70.239 (Sat Jul 13 02:41:53 2013)
80.82.70.239 (Sat Jul 13 02:41:55 2013)
80.82.70.239 (Sat Jul 13 02:41:56 2013)
80.82.70.239 (Sat Jul 13 02:41:57 2013)
80.82.70.239 (Sat Jul 13 02:41:58 2013)
80.82.70.239 (Sat Jul 13 02:41:59 2013)
80.82.70.239 (Sat Jul 13 02:42:00 2013)
80.82.70.239 (Sat Jul 13 02:42:02 2013)
80.82.70.239 (Sat Jul 13 02:42:03 2013)
80.82.70.239 (Sat Jul 13 02:42:04 2013)
80.82.70.239 (Sat Jul 13 02:42:05 2013)
80.82.70.239 (Sat Jul 13 02:42:06 2013)
80.82.70.239 (Sat Jul 13 02:42:07 2013)
80.82.70.239 (Sat Jul 13 02:42:09 2013)
80.82.70.239 (Sat Jul 13 02:42:10 2013)
80.82.70.239 (Sat Jul 13 02:42:11 2013)
80.82.70.239 (Sat Jul 13 02:42:12 2013)
80.82.70.239 (Sat Jul 13 02:42:13 2013)
80.82.70.239 (Sat Jul 13 02:42:15 2013)
80.82.70.239 (Sat Jul 13 02:42:16 2013)
80.82.70.239 (Sat Jul 13 02:42:17 2013)
80.82.70.239 (Sat Jul 13 02:42:18 2013)
80.82.70.239 (Sat Jul 13 02:42:19 2013)
80.82.70.239 (Sat Jul 13 02:42:20 2013)
80.82.70.239 (Sat Jul 13 02:42:22 2013)
80.82.70.239 (Sat Jul 13 02:42:23 2013)
80.82.70.239 (Sat Jul 13 02:42:24 2013)
80.82.70.239 (Sat Jul 13 02:42:25 2013)
Date template hits:
0 hit(s): MONTH Day Hour:Minute:Second
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
0 hit(s): Year/Month/Day Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
70 hit(s): Day/MONTH/Year:Hour:Minute:Second
0 hit(s): Month/Day/Year:Hour:Minute:Second
0 hit(s): Year-Month-Day Hour:Minute:Second
0 hit(s): Year.Month.Day Hour:Minute:Second
0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
0 hit(s): Day-Month-Year Hour:Minute:Second
0 hit(s): TAI64N
0 hit(s): Epoch
0 hit(s): ISO 8601
0 hit(s): Hour:Minute:Second
0 hit(s): <Month/Day/Year@Hour:Minute:Second>
Success, the total number of match is 29
However, look at the above section 'Running tests' which could contain important
information.
As you can see, I have a failregex set up in a filter and the filter is enabled. Using fail2ban-regex, the filter does find a match in the log file I'm monitoring. (I'm actively being hit by a problematic IP address right now which is making testing quite easy.)
So now I restart fail2ban and observe the rules taking effect:
$ sudo service fail2ban restart
* Restarting authentication failure monitor fail2ban [ OK ]
$ tail /var/log/fail2ban.log -n 50
2013-07-13 02:42:58,014 fail2ban.server : INFO Stopping all jails
2013-07-13 02:42:58,745 fail2ban.jail : INFO Jail 'apache-xmlrpc' stopped
2013-07-13 02:42:59,439 fail2ban.jail : INFO Jail 'ssh' stopped
2013-07-13 02:42:59,440 fail2ban.server : INFO Exiting Fail2ban
2013-07-13 02:43:08,055 fail2ban.server : INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.6
2013-07-13 02:43:08,057 fail2ban.jail : INFO Creating new jail 'ssh'
2013-07-13 02:43:08,111 fail2ban.jail : INFO Jail 'ssh' uses Gamin
2013-07-13 02:43:08,397 fail2ban.filter : INFO Added logfile = /var/log/auth.log
2013-07-13 02:43:08,404 fail2ban.filter : INFO Set maxRetry = 6
2013-07-13 02:43:08,414 fail2ban.filter : INFO Set findtime = 600
2013-07-13 02:43:08,435 fail2ban.actions: INFO Set banTime = 600
2013-07-13 02:43:09,277 fail2ban.jail : INFO Creating new jail 'apache-xmlrpc'
2013-07-13 02:43:09,277 fail2ban.jail : INFO Jail 'apache-xmlrpc' uses Gamin
2013-07-13 02:43:09,283 fail2ban.filter : INFO Added logfile = /var/log/apache2/other_vhosts_access.log
2013-07-13 02:43:09,286 fail2ban.filter : INFO Set maxRetry = 6
2013-07-13 02:43:09,289 fail2ban.filter : INFO Set findtime = 600
2013-07-13 02:43:09,292 fail2ban.actions: INFO Set banTime = 600
2013-07-13 02:43:09,458 fail2ban.jail : INFO Jail 'ssh' started
2013-07-13 02:43:09,792 fail2ban.jail : INFO Jail 'apache-xmlrpc' started
2013-07-13 02:43:11,361 fail2ban.actions: WARNING [apache-xmlrpc] Ban 80.82.70.239
$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
244 39277 fail2ban-apache-xmlrpc tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
101 7716 fail2ban-ssh tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * * 0.0.0.0/0 127.0.0.0/8 reject-with icmp-port-unreachable
3404 582K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
349 20900 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
12 720 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
2 80 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 5/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
2 80 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3331 4393K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain fail2ban-apache-xmlrpc (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 80.82.70.239 0.0.0.0/0
244 39277 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain fail2ban-ssh (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 223.4.147.8 0.0.0.0/0
101 7716 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
As the fail2ban log shows, the rule set appears to be configured correctly. You can see already that the problematic IP address is being caught right away and banned. The output of iptables shows that it is in fact being dropped.
Already however, I'm observing that there are no dropped packets for that IP address that matches under the fail2ban-apache-xmlrpc chain. Sure enough, I check the apache logs:
$ tail /var/log/apache2/other_vhosts_access.log
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:53 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:54 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:56 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:57 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:58 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:59 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:44:00 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:44:02 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
Nope, it's not getting blocked! I can also confirm this in the fail2ban log:
$ tail /var/log/fail2ban.log
2013-07-13 02:52:30,757 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:37,767 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:44,783 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:51,814 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:58,830 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:05,842 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:11,858 fail2ban.actions: WARNING [apache-xmlrpc] Unban 80.82.70.239
2013-07-13 02:53:12,910 fail2ban.actions: WARNING [apache-xmlrpc] Ban 80.82.70.239
2013-07-13 02:53:20,118 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:27,129 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
It keeps reappearing in the apache log and thus fail2ban is trying to keep banning it!
I quite honestly cannot figure out for the life of me why iptables is not dropping the traffic from this IP address. The rule order seems correct to me, with the DROP coming before anything else.
I've Google a bunch of results where people are having a similar issue, but it always seems to come back to a problem banning SSH traffic where they are on a nonstandard port. In my case I'm just trying to ban an IP address on standard http port 80.
I hope I'm just overlooking something insanely simple. This is a VPS running Ubuntu 12.04 on Linode. If anyone has any ideas, please let me know. Many thanks…
EDIT: Here is the output of iptables -S
$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-apache-xmlrpc
-N fail2ban-ssh
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-apache-xmlrpc
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j ACCEPT
-A fail2ban-apache-xmlrpc -s 80.82.70.239/32 -j DROP
-A fail2ban-apache-xmlrpc -j RETURN
-A fail2ban-ssh -s 223.4.147.8/32 -j DROP
-A fail2ban-ssh -j RETURN
Best Answer
The
iptables -soutput looks correct and I don't know how80.82.70.239/32is getting toany:80on your server through the firewall. My first guess is that you have a proxy / load balancer in front of the server and Apache is logging theHTTP_X_FORWARDED_FORheader or what ever that is called. If that turns out to be the case you will have to move your firewall logic to the proxy / load balancer or down to the application level (Apache mathing theFORWARDED_FORheader and deny access.Either way:
The next course of action that I would take, is to capture the output of
iptables -syou posted above. Disable fail2ban and load the configuration with the fail2ban chain and IP address blocked into iptables.But do so with the following as the first
-Arule:If you would feel better trapping 80 and 443 go for it. The thought is that the logs from the FIREWALL may show something we are missing if we pay attention to the packets from suspect sources.