Nginx – AWStats log parsing – custom Nginx log format

awstatsnginx

I've installed AWStats 7.0 (the latest version in the Amazon Linux repository) to try to get additional information about bandwidth usage. I'm having trouble getting AWStats to parse my logs – I suspect it's because I can't get the LogFormat right.

I've tried many variations and I just can't get it working.

Here's my Nginx log format

log_format  main  '$remote_addr - $remote_user [$time_local] "$host" "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for" "$request_time" '
                  '"$upstream_cache_status" "$sent_http_content_encoding" ';

Here's a log entry

1.1.1.1 - - [12/Mar/2017:07:23:53 +1300] "www.example.com" "GET /url/ HTTP/1.1" 200 7455 "https://www.google.ru/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" "46.71.136.54" "0.000" "HIT" "gzip"

Here's my AWStats configuration file. Anything not here is standard and inherited from the main configuration file

# Path to you nginx vhost log file
LogFile="/var/log/nginx/pts.access.log"

# Domain of your vhost
SiteDomain="example.com"

# Directory where to store the awstats data
DirData="/var/lib/awstats/pts/"

# Other alias, basically other domain/subdomain that's the same as the domain above
HostAliases="www.example.com"

LogFormat = "%host %logname %time1 %virtualname %methodurl %code %bytesd %refererquot %uaquot %otherquot %otherquot %otherquot %otherquot"

Here's the awstats output

[root]# /usr/share/awstats/tools/awstats_updateall.pl now -awstatsprog=/usr/share/awstats/wwwroot/cgi-bin/awstats.pl
Running '"/usr/share/awstats/wwwroot/cgi-bin/awstats.pl" -update -config=example.com -configdir="/etc/awstats"' to update config example.com
Create/Update database for config "/etc/awstats/awstats.example.com.conf" by AWStats version 7.0 (build 1.971)
From data in log file "/var/log/nginx/pts.access.log"...
Phase 1 : First bypass old records, searching new record...
Searching new records from beginning of log file...
Jumped lines in file: 0
Parsed lines in file: 323
 Found 323 dropped records,
 Found 0 comments,
 Found 0 blank records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

Can anyone spot what's not right? I can't find any additional information or awstats logs that would give further information.

Best Answer

One possible issue is here:

log_format  main  '$remote_addr - $remote_user [$time_local]...

Corresponding configuration in AWStats:

LogFormat = "%host %logname %time1

And your log file contains:

1.1.1.1 - - [12/Mar/2017:07:23:53 +1300]

%logname matches to only a single string, that is, the username provided in HTTP authentication. Now, your log file contains two dashes, first one from your configuration, and the second one means an empty username.

So, AWStats tries to interpret the second dash as a timestamp, and that causes it to consider the record as failed.

So, you either need to add the dash to AWStats log format string, or remove the dash from nginx log format.

As a side note, you don't need to quote your last parameters ($request_time, $upstream_cache_status, $sent_http_content_encoding) in nginx log, since they cannot contain spaces.

You can also use %extraX in AWStats configuration if you want to use that information in building reports based on those facts.

Related Topic