Restarting rsyslog re-sends logs again

rsyslogubuntu-12.04

I am running Ubuntu 12.04.1 LTS on EC2. I have a bunch of application servers which are configured to forward their logs to a central server via rsyslog.

Since putting in Nagios monitoring on the log files on the central server, I've been getting alerts indicating that particular application servers are failing to forward their logs to the centralized server.

Logging into the machines and restarting the rsyslog service fixes the problem. However, rsyslog then re-transmits the logs again, resulting in duplicates on the collector. Why is it doing this?

Best Answer

The freezing of log transmission aspect of this problem appears to likely be due to a bug in the way rsyslog is configured out of the box with 'buntu: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/940030

I was also able to find an interesting post on expertsexchange: www.expert-sex-change.com/OS/Linux/Administration/Q_27511414.html:

> I recently ran into a problem and was hoping someone could help me figure it out.  I am
> running Ubuntu 10.04 and rsyslog 5.8.1.  I setup rsyslog to monitor a few files and send
> the any changes to those files to Loggly (great service by the way).  Well, it all works
> great, except for the fact that whenever I restart rsyslog it resends everything from those
> files that it had sent in the past.  Now, it is my understanding that rsyslog should use
> the $WorkDirectory and create what file that you set for $InputFileStateFile to save the
> state of the monitored file so that it does not send duplicate entries.
> 
> Here is an example that I'm working with:
> 
> $WorkDirectory /var/log/rsyslog
> 
> # Monit log file
> $InputFileName /var/log/monit.log
> $InputFileTag monit
> $InputFileStateFile monit-state
> $InputFileSeverity info
> $InputRunFileMonitor
> 
> So, from my description above, rsyslog should create a "monit-state" file in 
> "/var/log/rsyslog" which would save the state of the /var/log/monit.log" file so no 
> duplicates are sent.  Is that correct?  If so, this is not happening!
> 
> What should the premission of "/var/log/rsyslog" be?  Do I have to touch "monit-state" in
> "/var/log/rsyslog" (I tried that already and nothing happened)?

With an interesting answer:

> Ok, I found this site (Trouble Shooting Rsyslog) and went through the interactive debugging
> mode using:
> 
> rsyslogd -c5 -dn > logfile
> 1:
> Select all
> Open in new window
> 
> 
> I was able to see that it was in fact a permissions problem.  I opened up permissions to
> 777 for my $WorkDirectory, just to test, and rsyslog did create the state files and
> everything worked as planned.  I will continue to play with the permissions so that it's
> not 777 (obviously), but I wanted to let anyone else that might be having this problem know
> how I solved it.

I have adjusted the permissions on /var/spool/rsyslog to be owned by user "syslog". I've tested for duplicate transmissions after applying this change and it seems to have fixed the problem. If the problem re-emerges or I find out that this didn't fix it, I will return here and update the answer.

SOLUTION:

$ sudo chown syslog /var/spool/rsyslog
Related Topic