Forwarding specific logs rsyslog

loggingrsyslog

TL;DR – send specific logs with rsyslog (to a redis server) : how to select the logs to be sent ?

I want to forward to a redis server a set (and only that set) of logs, say for instance nginx logs in /var/log/nginx/*.log. For this, I was thinking of using a rsyslog facility (local7 in my example). However :

  • I cannot process only the local7.* logs to the redis server : I receive all the logs of the system (auth, authpriv, cron, local7 as well, etc.)
  • I cannot process all the logs of one directory (e.g.: /var/log/nginx/*.log won't work but /var/log/nginx/some-access.log will be sent to my redis server by rsyslog. How to get all the logs of one directory then?)

Configuration has three modules in use and sends logs from local7 to my redis server this way :

local7.* @redis_ip:port
$ModLoad imuxsock # provides support for local system logging
$ModLoad omhiredis # support for sending to Redis
$ModLoad imfile # For tailing files

The two other blocs of code are two different ways to configure rsyslog I came across.

Config 1 (common config people suggest):

$InputFileName /var/log/nginx/*.log
$InputFileTag nginx
$InputFileFacility local7
$InputRunFileMonitor

Config 2 (different syntax – the one I found on rsyslog's documentation for version 8.16.0) :

input(
  type="imfile"
  File="/var/log/nginx/*.log"
  Tag="nginx:"
  Facility="local7"
)

To output to redis :

action(
  name="rsyslog_redis"
  type="omhiredis"
  mode="queue"
  key="rsyslog_redis_key"
  template="jsonlines"  # use a JSON template defined below
)
  • Rsyslog 8.16.0, build from sources with module omhiredis (for output to Redis)
  • Debian 8

Note

If I simply remove config 1 or config 2 and use authpriv.* @redis_ip:port for instance, I will still get all the logs (so logs from facility syslog, cron, auth, authpriv, etc.) as if authpriv.* in authpriv.* @redis_ip:port had no impact on rsyslog.

I start rsyslog with /usr/local/sbin/rsyslog -f /etc/rsyslog.conf and checking it with option -N1 says it is all correct.

The questions I've checked haven't changed anything for me :

Best Answer

Since I finally got something working and probably that I know where was my mistake, here is an answer to my own question :

$ModLoad imuxsock # provides support for local system logging
$ModLoad omhiredis # support for sending to Redis
$ModLoad imfile # For tailing files

if $syslogtag == "nginx:" then {
  action(
    name="rsyslog_redis"
    server="redis_ip"
    port="redis_port"
    type="omhiredis"
    mode="queue"
    key="rsyslog_redis" # we need the same key in Logstash's config
    template="jsonlines"  # use the JSON template defined below
  )
}

input(
  type="imfile"
  File="/var/log/nginx/access.log"
  Tag="nginx:"
)

allows me to send nginx access.log and only those to my redis server.

My error came from the block action( .. ) that wasn't limited by the if statement. It was therefore sending all the logs. That explains my first Note in the question where authpriv.* @redis_ip:port wouldn't change anything.

Answer : action( .. ) will send logs, even though no server is defined in it, so surround it by if statements to chose which logs gets forwarded.

Note : I haven't added the template "jsonlines" to the config as one can easily find it on internet and that it takes some unnecessary space here.

Related Topic