Linux – How todentify an anonymous process that’s changing a system setting

Linux auditing can help. It will at least locate users and processes making datagram network connections. UDP packets are datagrams.

First, install the auditd framework on your platform and ensure that auditctl -l returns something, even if it says that no rules are defined.

Then, add a rule to watch the system call socket() and tag it for easy finding later (-k). I need to assume that you are on a 64-bit architecture, but you can substitute b32 in place of the b64 if you aren't.

auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET

You have to pick through man pages and header files to build this, but what it captures is essentially this system call: socket(PF_INET, SOCK_DGRAM|X, Y), where the third parameter is unspecified but frequently zero. PF_INET is 2 and SOCK_DGRAM is 2. TCP connections would use SOCK_STREAM which would set a1=1. (SOCK_DGRAM in the second parameter may be ORed with SOCK_NONBLOCK or SOCK_CLOEXEC, hence the &= comparison.) The -k SOCKET is our keyword we want to use when searching audit trails later. It can be anything, but I like to keep it simple.

Let a few moments go by and review the audit trails. Optionally, you could force a couple of packets by pinging a host out on the net, which will cause a DNS lookup to occur, which uses UDP, which should trip our audit alert.

ausearch -i -ts today -k SOCKET

And output similar to the section below will appear. I'm abbreviating it to highlight the important parts

type=SYSCALL ... arch=x86_64 syscall=socket success=yes exit=1 a0=2 a1=2 ... pid=14510 ... auid=zlagtime uid=zlagtime ... euid=zlagtime ... comm=ping exe=/usr/bin/ping key=SOCKET

In the above output, we can see that the ping command caused the socket to be opened. I could then run strace -p 14510 on the process, if it was still running. The ppid (parent process ID) is also listed in case it is a script that spawns the problem child a lot.

Now, if you have a lot of UDP traffic, this isn't going to be good enough and you'll have to resort to OProfile or SystemTap, both of which are currently beyond my expertise.

This should help narrow things down in the general case.

When you are done, remove the audit rule by using the same line you used to create it, only substitute -a with -d.

auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET

Check the trace of systemd process for mmap/mmunmap calls. It should reveal the problem:

yum install strace
strace -ff -p 1

It's a quick and dirty way to diagnose memory leaks. Strace of systemd process should look similiar:

recvmsg(23, {msg_name(0)=NULL, msg_iov(1)=[{"WATCHDOG=1", 4096}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS{pid=620, uid=0, gid=0}}, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 10
open("/proc/620/cgroup", O_RDONLY|O_CLOEXEC) = 20
fstat(20, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfd734e000
read(20, "10:cpuset:/\n9:perf_event:/\n8:hug"..., 1024) = 164
close(20)                               = 0
munmap(0x7fcfd734e000, 4096)            = 0

It allocates memory, does something, than releases memory.
Checking the trace of system calls systemd does, you should discover where it can't finish calls and release allocated memory.
I suppose there's a problem with improperly mounted pseudo-filesystems or selinux, so systemd can't finish its calls.