Nginx – Logwatch configured for nginx with custom log format gives empty output

centoscentos5logwatchnginx

Problem

I have configured logwatch (CentOS 5.8, x64) to include nginx, using this as a guideline and using the Apache and nginx documentation on log formats. The problem is, that I'm using a specific log format, being:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $scheme:$server_port '
                  '$status $body_bytes_sent "$http_referer" '
                  'Upstream ["$upstream_addr" ($upstream_response_time) $upstream_status : $upstream_cache_status] '
                  '"$http_user_agent" "$http_x_forwarded_for"';

(from /etc/nginx/nginx.conf)

I have translated this log format into:

$LogFormat "%h %l %u %t \"%r\" %H:%p %>s %b \"%{Referer}i\" Upstream [\"%{Upstream-address}e\" (%{Upstream-response-time}e) %{Upstream-status}e : %{Upstream-cache-status}e] \"%{User-Agent}i\" \"%{X-Forwarded-For}e\""

for Logwatch. While studying /usr/share/logwatch/scripts/services/http, I found that anything %{...}e that is not predefined, will be ignored, so I thought this would be the best way to include these upstream-variables.

Logwatch doesn't give any output, though, considering nginx.

What I've done

I have created the following logwatch files:
/usr/share/logwatch/default.conf/logfiles/nginx.conf:

########################################################
#   Define log file group for nginx
# http://8bitpipe.com/?p=516
########################################################

# What actual file?  Defaults to LogPath if not absolute path....
LogFile = nginx/*access.log


# If the archives are searched, here is one or more line
# (optionally containing wildcards) that tell where they are...
#If you use a "-" in naming add that as well -mgt
Archive = nginx/archive/*access.log*

# Expand the repeats (actually just removes them now)
*ExpandRepeats

# Keep only the lines in the proper date range...
*ApplyhttpDate

/usr/share/logwatch/default.conf/services/nginx.conf:

###########################################################################
# Configuration file for nginx filter
###########################################################################

Title = "nginx"

# Which logfile group...
LogFile = nginx

# Define the log file format
#
# This is now the same as the LogFormat parameter in the configuration file
# for httpd.  Multiple instances of declared LogFormats in the httpd
# configuration file can be declared here by concatenating them with the
# '|' character.  The default, shown below, includes the Combined Log Format,
# the Common Log Format, and the default SSL log format.
#$LogFormat = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"|%h %l %u %t \"%r\" %>s %b|%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
$LogFormat "%h %l %u %t \"%r\" %H:%p %>s %b \"%{Referer}i\" Upstream [\"%{Upstream-address}e\" (%{Upstream-response-time}e) %{Upstream-status}e : %{Upstream-cache-status}e] \"%{User-Agent}i\" \"%{X-Forwarded-For}e\""

# The following is supported for backwards compatibility, but deprecated:
# Define the log file format
#
#   the only currently supported fields are:
#                       client_ip
#                       request
#                       http_rc
#                       bytes_transfered
#                       agent
#
#$HTTP_FIELDS = "client_ip ident userid timestamp request http_rc bytes_transfered referrer agent"
#$HTTP_FORMAT = "space     space space    brace    quote   space        space       quote   quote"
# Define the field formats
#
#   the only currently supported formats are:
#                       space = space delimited field
#                       quote = quoted ("..") space delimited field
#                       brace = braced ([..]) space delimited field

# Flag to ignore 4xx and 5xx error messages as possible hack attempts
#
# Set flag to 1 to enable ignore
# or set to 0 to disable
$HTTP_IGNORE_ERROR_HACKS = 0

# Ignore requests
# Note - will not do ANY processing, counts, etc... just skip it and go to
# the next entry in the log file.
# Examples:
# 1. Ignore all URLs starting with /model/ and ending with 1 to 10 digits
#   $HTTP_IGNORE_URLS = "^/model/\d{1,10}$"
#
# 2. Ignore all URLs starting with /model/ and ending with 1 to 10 digits and
#   all URLS starting with /photographer and ending with 1 to 10 digits
#   $HTTP_IGNORE_URLS = "^/model/\d{1,10}$|^/photographer/\d{1,10}$"
#   or simply:
#   $HTTP_IGNORE_URLS = "^/(model|photographer)/\d{1,10}$"
#
# vi: shiftwidth=3 tabstop=3 et

And I have symlinked /usr/share/logwatch/scripts/services/http to /usr/share/logwatch/scripts/services/nginx.

This does not give any error when executing logwatch, but it also doesn't give any output, while there are definitely logfiles to parse.

Executing logwatch --service nginx --print --range All --debug 7 gives, for example:

** lot of blabla about config files **

export LOGWATCH_DATE_RANGE='all'
export LOGWATCH_OUTPUT_TYPE='unformatted'
export LOGWATCH_TEMP_DIR='/var/cache/logwatch/logwatch.vdVyg9y2/'
export LOGWATCH_DEBUG='7'

Preprocessing LogFile: nginx
'/var/log/nginx/www.xxxx1.org-access.log' '/var/log/nginx/www.xxxx2.com-access.log' '/var/log/nginx/www.xxxx3.com-access.log' '/var/log/nginx/www.xxxx4.com-access.log' '/var/log/nginx/www.xxxx5.com-access.log' '/var/log/nginx/www.xxxx6.com-access.log' '/var/log/nginx/www.xxxx7.com-access.log' '/var/log/nginx/www.xxxx8.com-access.log' '/var/log/nginx/www.xxxx9.com-access.log' '/var/log/nginx/www.xxxx10.com-access.log' '/var/log/nginx/www.xxxx11.com-access.log' '/var/log/nginx/www.xxxx12-access.log' '/var/log/nginx/www.xxxx13.nu-access.log' '/var/log/nginx/www.xxxx14.org-access.log'   2>/dev/null | /usr/bin/perl /usr/share/logwatch/scripts/shared/expandrepeats ''| /usr/bin/perl /usr/share/logwatch/scripts/shared/applyhttpdate ''>/var/cache/logwatch/logwatch.vdVyg9y2/nginx

TimeFilter: Period is all

TimeFilter: SearchDate is (../.../....:..:..:..)

TimeFilter: Debug SearchDate is ( / / )
DEBUG: Inside ApplyHTTPDate...
DEBUG: Looking For: (../.../....:..:..:..)
export http_ignore_error_hacks='0'
export logformat "%h %l %u %t \"%r\" %h:%p %>s %b \"%{referer}i\" upstream [\"%{upstream-address}e\" (%{upstream-response-time}e) %{upstream-status}e : %{upstream-cache-status}e] \"%{user-agent}i\" \"%{x-forwarded-for}e\""=''

Processing Service: nginx
 ( cat /var/cache/logwatch/logwatch.vdVyg9y2/nginx  |  /usr/bin/perl /usr/share/logwatch/scripts/services/nginx) 2>&1

Why am I not getting any output?

Best Answer

I've noticed that logwatch will not generate a http/nginx log summary if the detail level is low or 0.

Use

  • --detail 1 for a summary of faulty requests,
  • --detail 5 or --detail med to also include a summary about the transferred amount of data and number of logged robots.
  • --detail 10 or --detail high to display the User-Agent of all logged robots.

Display nginx/http summary with detail level 0

Since I wanted logwatch to always display the full http log but not bloat all other logs I created an individual copy of the http script in /etc/

sudo cp /usr/share/logwatch/scripts/services/http /etc/logwatch/scripts/services/

and changed

my $detail = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;

to

my $detail = 10;

at the beginning of the /etc/logwatch/scripts/services/http script.

Please note that the individual script won't receive any automatic updates if the logwatch package will provide a new version of /usr/share/logwatch/scripts/services/http.