Linux – Rsyslog creating duplicate entries when forwarding application logs

linuxrsyslogsyslog

I'm trying to add a new application log forward to a host which is already setup to send it's logs to a remote syslog server. I'm following the directions here: http://www.rsyslog.com/doc/v7-stable/configuration/modules/imfile.html

The log file is written by an application and doesn't use any of the syslog facilities.

On the sender, I've added these two lines to /etc/rsyslog.conf:

$ModLoad imfile
$InputFilePollInterval 10

I've added a file in /etc/rsyslog.d/applogger.conf with the contents below.

input(type="imfile"
      File="/var/log/applog"
      Tag="applogger"
      StateFile="statefile2")

The logs are forwarded to the central logging system OK, but they are also being replicated into /var/log/syslog and /var/log/messages on the sending host, cluttering up the logs with lots of extra messages. The link above mentions duplicate entries but this is in regard to unique filenames for the applogs. This is the only file on the system with this name.

I must use rsyslog for this, so simply replacing it with syslog-ng is not an option.

How can I forward just the entries in /var/log/applog without duplicating the entries in the other logs?

Best Answer

Needed to add this line to applogger.conf. Any unique keyword found in the applog message entry will work.

:msg, contains, "appname" stop

Related Solutions

Linux – Rsyslog stops sending data to remote server after log rotation

The problem was actually coming from logrotate.

Basically with my configuration, running unicorn, I don't need to use the copytruncate directive. (which is what causes problems here)

USR1 - Reopen all logs owned by the worker process. See Unicorn::Util.reopen_logs for what is considered a log. Log files are not reopened until it is done processing the current request, so multiple log lines for one request (as done by Rails) will not be split across multiple logs.

This started working properly after updating to this configuration:

/home/user/my_app/shared/log/*.log {
  daily
  missingok
  dateext
  rotate 30
  compress
  notifempty
  extension gz
  create 640 user user
  sharedscripts

  post-rotate
    # Telling Unicorn to reload files
    test -s /home/user/my_app/shared/pids/unicorn.pid && kill -USR1 "$(cat /home/user/my_app/shared/pids/unicorn.pid)"

    # Reloading rsyslog telling it that files have been rotated
    reload rsyslog 2>&1 || true
  endscript
}

(rsyslog) Forwarding a specific log only

It's showing up twice because you're sending it twice.

The "*.* @address:514" is sending everything that this syslog (client) receives as input, which means both normal logs received by the syslog subsystem (i.e. read from /dev/log), and by reading /var/log/secure after it's been written to by syslog itself.

What you probably want to do is add: $InputFileFacility local0 (or some other locally unused localX facility) then change:

*.* @address:514

to local0.* @address:514

This will then ensure you are only forwarding to @address log lines that are read from /var/log/secure, with the additional tag that you want (assuming that the tag is an important thing you really want to see on the remote syslog server)

If the extra tag is unimportant, you simply don't need the InputFile* directives, and only need the forwarding (*.* @address:514). If you want to be selective, change *.* to just the logs you're interested in forwarding.

Related Topic