I wasn't clear if you configured syslog on all of your clients or not. Your tcpdump seems to show inbound syslog traffic, but you might want check that you aren't simply seeing traffic from one remote syslog client. Be sure that Syslogd is configured on each client machine as well. You need to add a line which looks something like this, where syslog.example.org is your syslog server:
# Log all messages to central syslog server
*.* @syslog.example.org
To verify this is working, use the logger(1) utility. I run a command like this on all of my machines in one window, often in a loop like for HOST in ALLOFMYHOSTS; do ssh $HOST 'logger'.....
logger -t stefantest stefantest1
Then in a separate window connected to the syslog, I do a tail -f /var/log/YOURLOGFILE |grep stefantest. This will confirm if syslog itself is working from each host or not. Then, go from there.
Linux auditing can help. It will at least locate users and processes making datagram network connections. UDP packets are datagrams.
First, install the auditd framework on your platform and ensure that auditctl -l returns something, even if it says that no rules are defined.
Then, add a rule to watch the system call socket() and tag it for easy finding later (-k). I need to assume that you are on a 64-bit architecture, but you can substitute b32 in place of the b64 if you aren't.
auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
You have to pick through man pages and header files to build this, but what it captures is essentially this system call: socket(PF_INET, SOCK_DGRAM|X, Y), where the third parameter is unspecified but frequently zero. PF_INET is 2 and SOCK_DGRAM is 2. TCP connections would use SOCK_STREAM which would set a1=1. (SOCK_DGRAM in the second parameter may be ORed with SOCK_NONBLOCK or SOCK_CLOEXEC, hence the &= comparison.) The -k SOCKET is our keyword we want to use when searching audit trails later. It can be anything, but I like to keep it simple.
Let a few moments go by and review the audit trails. Optionally, you could force a couple of packets by pinging a host out on the net, which will cause a DNS lookup to occur, which uses UDP, which should trip our audit alert.
ausearch -i -ts today -k SOCKET
And output similar to the section below will appear. I'm abbreviating it to highlight the important parts
type=SYSCALL ... arch=x86_64 syscall=socket success=yes exit=1 a0=2 a1=2 ... pid=14510 ... auid=zlagtime uid=zlagtime ... euid=zlagtime ... comm=ping exe=/usr/bin/ping key=SOCKET
In the above output, we can see that the ping command caused the socket to be opened. I could then run strace -p 14510 on the process, if it was still running. The ppid (parent process ID) is also listed in case it is a script that spawns the problem child a lot.
Now, if you have a lot of UDP traffic, this isn't going to be good enough and you'll have to resort to OProfile or SystemTap, both of which are currently beyond my expertise.
This should help narrow things down in the general case.
When you are done, remove the audit rule by using the same line you used to create it, only substitute -a with -d.
auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
Best Answer
I just did some testing, and while the port is definitely opened by syslogd, it doesn't look like it's actually handling or logging any activity directed to it on UDP 514. You can verify this by sending data with
netcat:And then checking the logfile. I tested it on two RHEL5 boxes, and if
-risn't used, it won't actually process the logs.Update: Another solution (or, really, work-around) that I just thought of would be to install rsyslog (or syslog-ng) as a replacement syslog daemon for the default sysklogd. Neither of these alternate syslog daemons suffer from the bug described above.
rsyslog is the default syslog daemon with RHEL 6.x, and is available as a supported package for RHEL/CentOS 5.2+. rsyslog is under active development (sysklogd is not, and hasn't been for years). rsyslog also supports many advanced features and functionality. As mentioned, with RHEL/CentOS 5.2+, switching from the stock syslogd to rsyslog is as easy as
yum install rsyslog.If you do decide on replacing your syslog daemon, and you want something cleaner and more flexible (in my opinion), Syslog-NG is worth taking a look at. The config file doesn't maintain backwards compatibility with the old
syslog.conf(rsyslog does), so it can seem a little complicated at first glance, but for complex or advanced logging setups (especially at a central loghost), Syslog-NG is an excellent choice.