Nginx – Separate Nginx access log file for certain requests only

blockingloggingnginxrequestUbuntu

As far as I can see Nginx supports by default 2 log files: error_log (tracks issues related to the Nginx server itself) and access_log (tracks requests processed by Nginx). Whilst it is possible to control the format of access_log with the log_format directive, I have been unsuccessful at finding a way of logging only certain requests to a separate file, and therefore would like to ask the question on SF as a reference for future readers:

Is there a way to log certain requests to a different log file than the one defined by access_log?

FYI the reason behind this question is that I have a rule that denies access to unwanted crawlers with a 200 (because 403 would give them a hint that they're being blocked), and filtering those requests out of the access_log becomes more difficult.

Best Answer

cjc put me on the right track. Using access_log in an if statement by itself is not possible (You get a nginx: [emerg] "access_log" directive is not allowed here error). So the workaround is as follows:

if ($http_user_agent ~* (crawler) ) {
  set $crawler 'yes';
}
location ~ .* {
  if ($crawler = 'yes') {
    access_log /var/log/nginx/blockedbots.log;
    return 200;
    }
}

Correct way in new versions of nginx

Turn out my first answer to this question was correct at certain time, but it turned into another pitfall - to stay up to date please check Taxing rewrite pitfalls

I have been corrected by many SE users, so the credit goes to them, but more importantly, here is the correct code:

server {
       listen         80;
       server_name    my.domain.com;
       return         301 https://$server_name$request_uri;
}

server {
       listen         443 ssl;
       server_name    my.domain.com;
       # add Strict-Transport-Security to prevent man in the middle attacks
       add_header Strict-Transport-Security "max-age=31536000" always; 

       [....]
}

Limit NGINX Access Log File Size and Enable Compression

/etc/logrotate.d/nginx

/var/log/nginx/access_log {
    rotate 7
    size 5k
    dateext
    dateformat -%Y-%m-%d
    missingok
    compress
    sharedscripts
    postrotate
        test -r /var/run/nginx.pid && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

Best Answer

Related Solutions

NGINX Redirect – How to Rewrite All HTTP Requests to HTTPS While Maintaining Sub-Domain

Correct way in new versions of nginx

Limit NGINX Access Log File Size and Enable Compression

Related Topic