Forwarding from rsyslog to syslog-ng over TCP not working (although packets are reaching server)

rsyslogsyslog-ng

We use syslog-ng on our central syslog server (syslog-ng-2.1.4-9.el5 on CentOS 5.9). We were happily sending logs using syslogd and rsyslog from a mixture of Linux and Solaris hosts over UDP until yesterday when it finally became clear to me that we're losing a significant number of entries (yes, I should have heeded all the warnings).

I'm trying to change to using TCP. I'm keen (at the moment) to stick with syslog-ng at the centre and rsyslog on the senders and my understanding is that this should work. The central syslog server has multiple virtual interfaces which are used to segment sets of logs by function (which is why the udp() and tcp() statements specify the IP address to bind to).

I enabled TCP listeners at the syslog-ng end (see extract from config file below) – netstat -l shows listeners on port 514. As a test I changed the forwarding clause on one host (CentOS 6.4 with rsyslog-5.8.10-6.el6.x86_64) from @unixlog to @@unixlog.
I see the packets arriving at the central server and packets going back (looking with tcpdump on unixlog) so I think I've eliminated issues with iptables however nothing appears in the output file. I just tried turning iptables off for a while to check this – same thing.

I haven't tried turning on debugging for syslog-ng because this is a busy server – my next step is likely to be setting up a test syslog-ng server and pointing a single host at it. Before I do that is there anything else I should be looking at? Do I need to change the format of forwarded messages? My reading of the Syslog-ng 2.x docs suggests that this should work without any changes. I've tried changing the compatibility level option that rsyslog is
called with. Was initially set to 5, I've tried 0 .. 4 and removing the parameter completely – no difference in behaviour.

Rsyslog.conf on sender (with comments and local files removed) …

$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
*.err;kern.debug;daemon.notice;mail.crit                @unixlog.ncl.ac.uk
local3.debug                                            @cmdloghost.ncl.ac.uk

Extract from rsyslog.conf on unixlog

options {
  long_hostnames(off);
  sync(0);
  create_dirs(yes);
};

destination d_syslog {
  file("/var/log/incoming/syslogs/$HOST/syslog.$YEAR$MONTH$DAY.log");
};

# unixlog is 10.8.232.202
source unixlog_net { udp(ip(10.8.232.202)); tcp(ip(10.8.232.202)); };

log {
  source(unixlog_net);
  destination(d_syslog);
};

Best Answer

Problem was tcpwrappers which was obvious once I set up a test syslog-ng instance and ran it with debug output turned on. Adding the following clause to /etc/hosts.allow did the trick

syslog-ng: 10.0.0.0/255.0.0.0

Also realised that we didn't handle syslog-ng's internal messages (which may well have told us what the problem was). Have now added the following lines to do this ...

# Deal with syslog-ng's own messages
source s_internal { internal(); };
destination d_internal {file("/var/log/syslog-ng.log"); };
log { source(s_internal); destination(d_internal); };
Related Topic