Amazon Linux rsyslog config override

amazon-amiamazon-web-servicesrsyslog

I am using Amazon Linux AMI release 2015.03, I am trying to configure rsyslog to redirect logs to a remote logging server (in this case, logstash) by dropping a conf file in /etc/rsyslog.d.

The remote logging configuration seems to work fine, but the logs are still written to /var/log/messages as well. Which is a problem because they are filling up the disk.

My conf file looks like this:

# Log docker generated log messages to logstash
:syslogtag, startswith, "docker" @x.x.x.x:5000    
& ~

After some investigation with some other systems, I found that the rsyslog.conf file that exists by default in Amazon Linux is including this directive at the very end of the conf file:

$IncludeConfig /etc/rsyslog.d/*.conf

Other systems I have used have this directive higher up in the config file.. definitely before the default log configs. It appears to me that the defaults cannot be overridden because of this. Am I missing something?

Best Answer

It appears to me that the defaults cannot be overridden because of this. Am I missing something?

Don't think of it as overriding defaults, but you are otherwise correct. In your file, the & ~ is discarding the log message preventing further processing. If this occurs above the built-in processing for /var/log/messages, the log message will be dropped and thus excluded from that file. Since your $IncludeConfig is occurring after, so is your drop statement.

You have two options;

Move the $IncludeConfig statement up.
Fix log rotation so that your disk does not fill.

I would personally prefer the latter because I like to keep logs locally (for a reasonable period of time) as well as forward them to a remote server for aggregation/retention. Having log messages available in /var/log/messages will make troubleshooting more convenient as well as easier for anyone else using the system due to this being the expected location.

Related Solutions

Remote rsyslog only writing logging data to /var/log/syslog and not custom logfiles

OK, solved it. I changed the config entry in my custom 22-remote.conf file in /etc/rsyslog.d/ from this:

:msg, contains, "AMAZONA-COMPUTERNAME1" /var/log/dbm/server-1.log
& ~
:msg, contains, "AMAZONA-COMPUTERNAME2" /var/log/dbm/server2.log
& ~

To this:

if $fromhost-ip == '10.11.12.12' then /var/log/dbm/server-1.log
& ~
if $fromhost-ip == '10.11.13.13' then /var/log/dbm/server2.log
& ~

Linux – Rsyslog stops sending data to remote server after log rotation

The problem was actually coming from logrotate.

Basically with my configuration, running unicorn, I don't need to use the copytruncate directive. (which is what causes problems here)

USR1 - Reopen all logs owned by the worker process. See Unicorn::Util.reopen_logs for what is considered a log. Log files are not reopened until it is done processing the current request, so multiple log lines for one request (as done by Rails) will not be split across multiple logs.

This started working properly after updating to this configuration:

/home/user/my_app/shared/log/*.log {
  daily
  missingok
  dateext
  rotate 30
  compress
  notifempty
  extension gz
  create 640 user user
  sharedscripts

  post-rotate
    # Telling Unicorn to reload files
    test -s /home/user/my_app/shared/pids/unicorn.pid && kill -USR1 "$(cat /home/user/my_app/shared/pids/unicorn.pid)"

    # Reloading rsyslog telling it that files have been rotated
    reload rsyslog 2>&1 || true
  endscript
}

Best Answer

Related Solutions

Remote rsyslog only writing logging data to /var/log/syslog and not custom logfiles

Linux – Rsyslog stops sending data to remote server after log rotation

Related Topic