Linux – Strange processes on Server consume CPU

glusterfslinux

I have noticed 15% CPU load on the Server which is currently offline. It has mounted GlusterFS volume over TCP. Looking through the top, it shown me it's glusterfs. After that I tried to figure what exactly is using it and I got this:

# lsof /storage/
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF                NODE NAME
find    16433 nobody  cwd    DIR   0,19     8192 9259265867489333824 /storage/200000/200000/200700/200704/08

Then:

# ps uax | grep find
root     16415  0.0  0.0   4400   724 ?        SN   06:34   0:00 /bin/sh /usr/bin/updatedb.findutils
root     16423  0.0  0.0   4400   336 ?        SN   06:34   0:00 /bin/sh /usr/bin/updatedb.findutils
nobody   16431  0.0  0.0  39524  1376 ?        SN   06:34   0:00 su nobody -s /bin/sh -c /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)' \) -prune -o -print0
nobody   16432  0.0  0.0   4400   616 ?        SN   06:34   0:00 sh -c /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)' \) -prune -o -print0
nobody   16433  0.3  0.0  13612  1532 ?        SN   06:34   0:38 /usr/bin/find / -ignore_readdir_race ( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o -type d -regex \(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\) ) -prune -o -print0

I killed 16432 and 16433 and CPU now is %0 again.

Can someone tell me anything about these ugly find commands? Is it possible it's cause by the other Server which also have this /storage mounted?

According monitoring, it happens every day at the same time.

Best Answer

That looks like it's part of the daily updatedb job that runs to update the databases used by the locate command.

You'll probably find it in /etc/cron.daily as mlocate or similar.

If you use ps -ef you get the PID (process) and PPID (parent PID) which can be used to track back. You probably would have seen that the processes you killed had PPIDs 16415, 16423.

Tools like pstree are handy for this kind of thing too.

pstree -p -H5295

gives output like this

  |-sshd(5291)---sshd(5294)---bash(5295)-+-more(6098)
  |                                      `-pstree(6097)
Related Topic