Block requests from bots by pattern in apache with mod_rewrite. mod_rewrite not working

.htaccessapache-2.2botnetmod-rewrite

I'm trying to block requests from bots by pattern glitter_calendar on apache.

Prehistory. My server is heavily accessed by bots. The request URI is …glitter_calendar.. The wordpress returns 404. The server even goes down once a week. I tried blocking them with Fail2ban. Without success. The server load doesn't go down and I get new hosts appearing in my iptables block-rule constantly.

In my .htaccess I am trying to add a block rule by pattern glitter_calendar:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{THE_REQUEST} ^.*(wpdffcontent)|(glitter_calendar)|(mp4:).* [NC]
RewriteRule ^(.*)$ - [F,L]
</IfModule>

<Files xmlrpc.php>
Order Deny,Allow
Deny from all
</Files>

Found similar questions. Links:

My mod rewrite doesn't seem to work. I get Error 404 Not Found inside wordpress, not by means of apache.

Module mod_rewrite is turned on:

# httpd -M
...
Loaded Modules:
...
rewrite_module (shared)
...

The virtual host configuration is managed with virtualmin. How do I fix this?

Part of httpd.conf:

...
LoadModule rewrite_module modules/mod_rewrite.so
...

<VirtualHost x.x.x.x]:80>
SuexecUserGroup "#505" "#504"
ServerName example.com
ServerAlias www.example.com
ServerAlias webmail.example.com
ServerAlias admin.example.com
ServerAlias autoconfig.example.com
ServerAlias autodiscover.example.com
DocumentRoot /home/example/public_html
ErrorLog /var/log/virtualmin/example.com_error_log
CustomLog /var/log/virtualmin/example.com_access_log combined
ScriptAlias /cgi-bin/ /home/example/cgi-bin/
ScriptAlias /awstats/ /home/example/cgi-bin/
ScriptAlias /AutoDiscover/AutoDiscover.xml /home/example/cgi-bin/autoconfig.cgi
ScriptAlias /Autodiscover/Autodiscover.xml /home/example/cgi-bin/autoconfig.cgi
ScriptAlias /autodiscover/autodiscover.xml /home/example/cgi-bin/autoconfig.cgi
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/example/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
AddType application/x-httpd-php .php
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/example/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/example/fcgi-bin/php5.fcgi .php5
</Directory>
<Directory /home/example/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.example.com
RewriteRule ^(.*) https://example.com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.example.com
RewriteRule ^(.*) https://example.com:10000/ [R]
RemoveHandler .php
RemoveHandler .php5
php_admin_value engine Off
FcgidMaxRequestLen 1073741824
Alias /dav /home/example/public_html
Alias /pipermail /var/lib/mailman/archives/public
<Location /dav>
DAV on
AuthType Basic
AuthName "example.com"
AuthUserFile /home/example/etc/dav.digest.passwd
Require valid-user
ForceType text/plain
Satisfy All
RemoveHandler .php
RemoveHandler .php5
RewriteEngine off
</Location>
<Files awstats.pl>
AuthName "example.com statistics"
AuthType Basic
AuthUserFile /home/example/.awstats-htpasswd
require valid-user
</Files>
RedirectMatch /cgi-bin/mailman/([^/\.]*)(.cgi)?(.*) https://example.com:10000/virtualmin-mailman/unauthenticated/$1.cgi$3
RedirectMatch /mailman/([^/\.]*)(.cgi)?(.*) https://example.com:10000/virtualmin-mailman/unauthenticated/$1.cgi$3
php_value memory_limit 32M
IPCCommTimeout 41
Redirect /mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
Redirect /.well-known/autoconfig/mail/config-v1.1.xml /cgi-bin/autoconfig.cgi
</VirtualHost>

Best Answer

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{THE_REQUEST} ^.*(wpdffcontent)|(glitter_calendar)|(mp4:).* [NC]
RewriteRule ^(.*)$ - [F,L]
</IfModule>

This would need to go before your existing WordPress mod_rewrite directives, otherwise it is never going to be processed. (The WordPress front controller rewrites every request before your directives get a chance to run.) However, your directives should be written as a one-liner instead:

RewriteRule (?:wpdffcontent|glitter_calendar|mp4:) - [F]

No need for the <IfModule> wrapper (in fact, this should be omitted). RewriteEngine On only needs to occur once in the file (anywhere). It's more efficient to check the URL in the RewriteRule pattern, no need for an additional RewriteCond directive here. And no need for all the capturing groups (you have 4 capturing groups in your initial directives which are just an unecessary waste of resources). And the L is not required when using the F flag (L is effectively implied).

Only include the NC flag if you specifically need to block GliTTer_CALendar and GLITTER_calendar as well as glitter_calendar, etc. If all the requests are for glitter_calendar (all lowercase) then that is all you need to block.

However, I think it would be preferable to use a combination of mod_setenvif and mod_authz_host (Apache 2.2) to block these requests. (The block with mod_rewrite could be overridden if you had additional .htaccess files using mod_rewrite.)

For example:

SetEnvIf Request_URI "(?:wpdffcontent|glitter_calendar|mp4:)" blockit

Order Deny,Allow
Deny from env=blockit

Logically, any blocking directives should be the first things in your .htaccess file. Followed by canonical/external redirects, then internal rewrites (the original WordPress directives). However, if you have access to the server config (which you appear to have), then these should all go in your server config and disable .htaccess (ie. AllowOverride None).

Related Topic