Apache – Blocking Access via User Agent String

blacklistintrusion-preventionuseragent

I've got a scripter who is using a proxy to attack a website I'm serving.

I've noticed that they tend to access the site via software with a certain common user agent string (i.e. http://www.itsecteam.com/en/projects/project1_page2.htm "Havij advanced sql injection software" with a user_agent string of Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) Havij). I'm aware that any cracking software worth it's salt will probably be able to modify it's user agent string, but I'm fine with the scripter having to deal with that feature at some point.

So, is there any software out there for automatically blocking access & permanently blacklisting by matching user agent strings?

Best Answer

you can deny access by BrowserMatch and Deny from SetEnvIf Example:

SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
<Directory "/var/www">
        Order Allow,Deny
        Allow from all
        Deny from env=bad_bot
</Directory>

To permanenly block them you have to write custom log file and use fail2ban for example to ban them with iptables

For example create LogFormat

LogFormat "%a %{User-agent}i" ipagent

Add logging to your vhost/server-wide

CustomLog /var/log/apache2/useragent.log ipagent

/etc/fail2ban/filter.d/baduseragent.conf

[Definition]
failregex = ^<HOST> Mozilla/4\.0 \(compatible; MSIE 7\.0; Windows NT 5\.1; SV1; \.NET CLR 2\.0\.50727\) Havij$

/etc/fail2ban/jail.conf

[apache-bad-user-agent]

enabled  = true
port     = 80,443
protocol = tcp
filter   = baduseragent
maxretry = 1
bantime  = 86400
logpath  = /var/log/apache2/useragent.log

Related Solutions

User agent blacklist recommended

Generally yes if it meets your requirements/needs, unless there are specific conflicts you can find with their definitions and/or approaches. The prudent route is to monitor very closely during its initial implementation to ensure it fits your needs without introducing any unanticipated/unexpected side-affects.

Blocking by user-agent string in httpd.conf not effective

Try this, and if it fails, try it in a .htaccess file...

   #Bad bot removal
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^useragent1 [OR]
   RewriteCond %{HTTP_USER_AGENT} ^useragent2 [OR]
   RewriteCond %{HTTP_USER_AGENT} ^useragent3
   RewriteRule ^(.*)$ http://website-you-want-to-send-bad-bots-to.com

Follow this pattern, and don't put an [OR] on the very last one.

EDIT: New solution:

If you want to block all (friendly) bots, make a file called "robots.txt" and put it in where your index.html is. Inside it, put this:

User-agent: *
Disallow: /

You'd still need to maintain a list like my original answer (above) to disallow the bots that ignore robots.txt.

Best Answer

Related Solutions

User agent blacklist recommended

Blocking by user-agent string in httpd.conf not effective

Related Topic