Nginx – Blocking Malicious Connections with Fail2ban

fail2bannginx

Recently there was a big increase in attacks on my server which made me switch to a more aggressive fail2ban blocking policy.
In attempt to reduce the number of incoming connections I also reconfigured nginx (for clearly malicious requests):

keepalive_requests 0;
keepalive_timeout 0;

return 444;

(...)

error_page 400 500 502 503 504 =444 /444.html;

location /444.html {
    return 444;
}

so that without a response the next connection or request isn't send immediately and nginx doesn't return anything useful for the attacker.

Since they were often scanning for ports I also added a filter in fail2ban which bans them for all ports and protocols, avoiding further connections:

enabled = true
maxretry = 1
bantime = 1d
port = all
banaction = %(banaction_allports)s[blocktype=DROP,protocol=all]

But now I'm noticing that nginx is collecting an increasing number of established connections without closing them. They are all connections that are blocked by fail2ban.

On one hand this effectively reduced the number of attempts (monitored before DROPed), on the other hand I have to restart nginx now and then to clear all these blocked but open connections.

I'm looking for a better way to accomplish this, drop these connections without service restart or some similar idea.
Using nginx/1.14.2 in combination with fail2ban/0.11.1 on Debian Buster.

Thanks in advance!

Edit 1:

To clarify: There are quite many FIN-WAIT-1, LAST-ACK but also ESTAB connections of IPs that are banned by fail2ban. The ESTAB connections seem to stay there quite long (maybe even until the ban is revoked?). My general idea was to prevent them from spending additional time on this IP by mimicking an absent host and to reduce my network resource usage on the server.

I tried conntrack -D -s <ip> as proposed in the link of @Danila Vershinin's comment but switched to ss -K dst <ip> as additional fail2ban action. But this only works on established connections and mysteriously does not remove all of them.

Best Answer

on the other hand I have to restart nginx now and then to clear all these blocked but open connections.

This doesn't look like nginx or fail2ban issue to me.

I guess there are not open but rather half-closed established connections. Normally they would disappear after some system timeout (related to your network subsystem configuration), but it could be indeed a problem if they become a lot.
You can try to reconfigure your network subsystem to better handle TIME-WAIT states, like adjust some sysctl values (keepidle, keepinittime, keepintvl), disable socket lingering, reuse or recycle TW sheduled connections, etc pp), for instance see Coping with the TCP TIME-WAIT state on busy Linux servers.

You could also try to change the way how you reject the connections in fail2ban for instance try REJECT --reject-with tcp-reset instead of DROP (see this comment for subject DROP vs. REJECT or my PoC FW.IDS-DROP-vs-REJECT).

I'm looking for a better way to accomplish this, drop these connections without service restart or some similar idea.

You can also extend the action of fail2ban's jail with tcpkill or cutter.
See https://github.com/fail2ban/fail2ban/issues/2107#issuecomment-379722469 for example.

Related Topic