Rsyslog not forwarding when running as service

logrotatelogstashrsyslog

I have 2 local VM's running CentOS 7 to simulate our production environment. 1 proxy server and 1 centralized (ELK) server. On the ELK server is running logstash to collect al syslogs from running instances.
The proxy server is provisioned with Rsyslog (and logrotate) via Ansible. Install syslog and logrotate. Copy configuration. Restart service. Really basic configuration, just a *.* @@IP.ADD.RE.SS:12345 in a 10-logstash.conf located in /etc/rsyslog.d/ and the rest is standard.
But, it doesn't seem to send logs to logstash. When I tail /var/log/messages i see output from the proxy container running on the instance. But none is forwarded to the logging server.

So I googled and found the DebuggingOnDemand option. When i stop the rsyslogd service and manually run sudo rsyslog -n it starts forwarding logs. When i run rsyslog -d (without sudo) I get notified it can't write the .pid file.
So my guess is, when it is ran as a service it does not have sufficient rights to work properly?
I cant manually start rsyslog without sudo. But i can start rsyslog as a service (sudo service rsyslog start) and confirmed that it is running.

So where is exactly the problem? Is it the process itself? Is it not able to read the log files? I couldn't imagine my configuration is wrong, as rsyslog is forwarding correctly when run manually.
I dont know what logging or output to provide, so if any is needed please ask and i will try to provide.

rsyslogd -N 10 output (with another configuration file which is confirmed working in production, same forwarder but a bit more options defined)

rsyslogd: version 7.4.7, config validation run (level 10), master config /etc/rsyslog.conf
rsyslogd: module 'imuxsock' already in this config, cannot be added [try http://www.rsyslog.com/e/2221 ]
rsyslogd: $WorkDirectory: /rsyslog/work can not be accessed, probably does not exist - directive ignored [try http://www.rsyslog.com/e/2181 ]
rsyslogd: End of config validation run. Bye.

Thanks in advance!

Best Answer

Definitely, this is related to SELinux, but because disabling SELinux is not the best choice I corrected the problem whit this:

sudo semanage port -d -t unreserved_port_t -p tcp 9092
sudo semanage port -a -t http_port_t -p tcp 9092

And then restart syslogd. You can see SELinux alerts with this command:

tail -f /var/log/audit/audit.log
Related Topic