Prevent Apache2 from logging robots and image requests

apache-2.2log-fileslogging

I run a high-traffic site and I need to reign in my Apache2 access logs a bit. One way to do this, I've read, is to limit the types of things Apache2 logs. The obvious choice would be, to limit 3 types of requests:

  1. Requests made by bots
  2. Requests for images, js files, and css (I'm only interested in the page request. Not ALL the assets)
  3. Requests made by localhost (the web server itself)

I am running the Apache2 mod setinf but it doesn't look like any of my changes take affect when I tail -f my access log. I'm also running cronolog (if that matters) as a log rotator. Here's what is in my VirtualHost conf:

# LogLevel warn

# Variable is_local_referrer = requests made by the server itself
SetEnvIfNoCase Referer "^http://*.mysite.com/" is_local_referrer

# Variable dont_log = standard file requests
SetEnvIf Request_URI ".(jpg|jpeg|png|css|gif|ico|js)$" dont_log

# Variable is_a_robot = user agent is a robot
SetEnvIfNoCase User-Agent Robot is_a_robot

ErrorLog "|/usr/bin/cronolog /home/me/public/mysite.com/log/error_%Y_%m.log"
CustomLog "|/usr/bin/cronolog /home/me/public/mysite.com/log/access_%Y_%m.log" combined env=!is_local_referrer|!dont_log|!is_a_robot

Again, when I tail /usr/bin/cronolog /home/me/public/mysite.com/log/access_X_X.log Apache2 is logging every single request.

Best Answer

I don't believe (and the documentation doesn't suggest) that you can make a CustomLog statement like env=!is_local_referrer|!dont_log|!is_a_robot, i.e., across several environmental variables.

You can have all of the SetEnvIf statments set the "dont_log" and then just say env=!dont_log, or you can have other SetEnvIf statements operate on the other env variables to set some other variable which will be used in the CustomLog directive, in case you're using, say, is_local_referrer elsewhere, e.g.:

SetEnvIfNoCase Referer "^http://*.mysite.com/" is_local_referrer
SetEnvIf Request_URI ".(jpg|jpeg|png|css|gif|ico|js)$" is_static_asset
SetEnvIfNoCase User-Agent Robot is_a_robot

SetEnvIf is_local_referrer .* dont_log
SetEnvIf is_static_asset .* dont_log
SetEnvIf is_a_robot .* dont_log

That ".*" is to fulfill the syntax requirements shown at http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html#setenvif and there may be a better way to do this, as I haven't tested this configuration. Maybe you can set, e.g., is_local_referrer=1 and then regex on 1 in the "dont_log" statements.

Related Topic