Linux – How to Control Noise Level in logwatch(8)

linuxlogwatchmonitoringsyslog

Our Linux systems run logwatch(8) utility by default. On a RedHat/CentOS/SL system, Logwatch is called by the /etc/cron.daily/ cronjob, which then sends a daily email with the results. These emails have a subject like:

Subject: Logwatch for $HOSTNAME

The problem is that by default these daily emails are too noisy and contain a lot of superfluous information (HTTP errors, daily disk usage, etc) which are already monitored by other services (Nagios, Cacti, central syslog, etc). For 100 systems, the email load is unbearable. People ignore the emails, which means that we may miss problems which are picked up by logwatch.

How can I reduce the amount of noise generated by logwatch, but still use logwatch to notify us of significant problems?

I'll post my own answer below, but I would like to see what others have done.

Note: I have a similar question regarding FreeBSD, at FreeBSD: periodic(8) is too noisy. How can I control the noise level?

Best Answer

Overall, the available documentation for Logwatch lacks adequate explanation and is often far too vague. I pieced together some useful examples, and have reduced the Logwatch noise by over 95%.

Here's what I have found.

Keep in mind that you can find some Logwatch documentation at /usr/share/doc/logwatch-*/HOWTO-Customize-LogWatch, and it contains a few useful examples.

  1. On RHEL/CentOS/SL, the default logwatch configuration is under /usr/share/logwatch/default.conf/logwatch.conf

    These settings can be overriden by placing your local configuration under /etc/logwatch/conf/logwatch.conf. Place the following in that file to tell logwatch to completely ignore services like 'httpd' and the daily disk usage checks:

    # Don't spam about the following Services
Service = "-http"
Service = "-zz-disk_space"

  2. Sometimes I don't want to completely disable logwatch for a specific service, I just want to fine tune the results to make them less noisy. /usr/share/logwatch/default.conf/services/*.conf contains the default configuration for the services. These parameters can be overridden by placing your local configuration under /etc/logwatch/conf/services/$SERVICE.conf. Unfortunately, logwatch's ability here is limited, and many of the logwatch executables are full of undocumented Perl. Your choice is to replace the executable with something else, or try to override some settings using /etc/logwatch/conf/services.

    For example, I have a security scanner which runs scans across the network. As the tests run, the security scanner generates many error messages in the application logs. I would like logwatch to ignore errors from my security scanners, but still notify me of attacks from other hosts. This is covered in more detail at Logwatch: Ignore certain IPs for SSH & PAM checks?. To do this, I place the following under /etc/logwatch/conf/services/sshd.conf:

    # Ignore these hosts
*Remove = 192.168.100.1
*Remove = X.Y.123.123
# Ignore these usernames
*Remove = testuser
# Ignore other noise. Note that we need to escape the ()
*Remove = "pam_succeed_if\(sshd:auth\): error retrieving information about user netscan.*

    "

  3. logwatch also allows you to strip out output from the logwatch emails by placing regular expressions in /etc/logwatch/conf/ignore.conf. HOWTO-Customize-LogWatch says:

    ignore.conf: This file specifies regular expressions that, when matched by the output of logwatch, will suppress the matching line, regardless of which service is being executed.

    However, I haven't had much luck with this. My requirements need a conditional statement, which is something like 'If there are security warnings due to my security scanner, then don't print the output. But if there are security warnings from my security scanner and from some bad guys, then print the useful parts-- The header which says "Failed logins from:", the IPs of the bad hosts, but not the IPs of scanners.'

  4. Nip it at the source (As suggested by @user48838). These messages are being generated by some application, and then Logwatch is happily spewing the results to you. In these cases, you can modify the application to log less.

    This isn't always desirable, because sometimes you want the full logs to be sent somewhere (to a Central syslog server, central IDS server, Splunk, Nagios, etc.), but you don't want logwatch to email you about this from every server, every day.

Related Topic