Linux – Random 403 Forbidden error (Apache 2.4)

apache-2.4linuxPHP

My Apache 2.4 (CentOS 7) server hosts multiple websites. Sometimes I get a 403 Forbidden (You don't have permission to access file X on this server) error when accessing any of the websites files (mostly php scripts). Hitting F5 for a few seconds will resolve it and load the page normally. The files have correct permissions. I also got mod_evasive enabled but the log does not show any IP being blocked at that time.

 Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 cloudflare_module (shared)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_prefork_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_scgi_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgi_module (shared)
 php5_module (shared)
 evasive20_module (shared)

Output of apachectl -M above. Any idea what could cause this?

Best Answer

I was having this exact problem, I am not sure if the below reason is what always causes this error(specifically in the described in the question way) but this was the case for me, so I just wanted to share my thoughts.

Debian 7 wheezy (7.7), apache 2.2.2

I was making a feature when the user should change the messages' status as read/unread, while clicking on a "link", when ajax was being sent to server, so, while testing - quickly clicking on that link to see if it works fine(so it wont be 2 concurrent ajax requests) I got this error

Forbidden

You don't have permission to access /messages on this server.

The strange part was that before that there were a few successful ajax requests with normal path including the domain name like http://example.com/messages/changeStatus/11. Which means the code was ok. But, on the other hand if I could just wait a few seconds and try again, it would work fine.

I had mod-security and mod-evasive installed, so after this error I found these last lines from /var/log/apache2/modsec_audit.log file.

--ba0f4035-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /messages/changeStatus/24
on this server.</p>
</body></html>

--ba0f4035-H--
Apache-Error: [file "mod_evasive20.c"] [line 246] [level 3] client denied by server configuration: /home/user_name/www/example/messages, referer: http://example.com/messages
Stopwatch: 1421177262896100 4724 (- - -)
Stopwatch2: 1421177262896100 4724; combined=10, p1=0, p2=0, p3=2, p4=0, p5=7, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.6 (http://www.modsecurity.org/).
Server: Apache/2.2.22 (Debian) PHP/5.4.36-0+deb7u1 mod_ssl/2.2.27 OpenSSL/1.0.1h

--ba0f4035-Z--

Doing some more search, as I found out the log's error in this article. https://www.atomicorp.com/wiki/index.php/Mod_evasive

so, mod evasive is the reason, because the default params of it are too sensitive, mainly in /etc/apache2/mods-available/mod-evasive.conf file by default I had these options

<ifmodule mod_evasive20.c>
   DOSHashTableSize 3097
   DOSPageCount  2
   DOSSiteCount  50
   DOSPageInterval 1
   DOSSiteInterval  1
   DOSBlockingPeriod  10
   DOSLogDir   /var/log/mod_evasive
   DOSEmailNotify  EMAIL@DOMAIN.com
   DOSWhitelist   127.0.0.1
</ifmodule>

as we learn from the above link

MODEV_DOSPageCount - This is the threshhold for the number of requests for the same page (or URI) per page interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list.

and

MODEV_DOSPageInterval - The interval for the page count threshhold; defaults to 1 second intervals.

So, according to default options if I would do 2 requests for the same url during 1 second it would show 403 error, and it was smth that happened to me: ASA I increased the number to 20, I was unable already replicate the error message.

on other hand

MODEV_DOSBlockingPeriod The blocking period is the amount of time (in seconds) that a client will be blocked for if they are added to the blocking list. During this time, all subsequent requests from the client will result in a 403 (Forbidden) and the timer being reset (e.g. another 10 seconds). Since the timer is reset for every subsequent request, it is not necessary to have a long blocking period; in the event of a DoS attack, this timer will keep getting reset

so, as we can see, after DOSBlockingPeriod time has passed the ip will be deleted from blacklist; as I am guessing this is the reason, that there are no banned IPs in logs, also that, when clicking F5 after a few seconds it works fine, as the blocked period has passed.

I also tested this with long blocking period and small page count values, mainly setting 1000 and 1 respectively. and After a 2-3 ajax requests it started showing 403 and did not go away after a few seconds.

Hope this will help someone.

Related Topic