Linux – Some processes using 100% CPU and hangs on select() call for some time

linux

So I have this strange scenario: there is a server this apache/passenger/Rails and some other services. For example top command itself takes 100% Cpu time (also iostat and ruby, other processes seems normal). At first I though that this is famous futex bug Linux futex_wait() bug (as processor is Xeon E5), but this is Centos 6.5 with 3.2.13-grsec-xxxx-grs-ipv6-64 kernel, and this seems to not affected by futex bug. Here is output of top command:

 top - 18:14:46 up 60 days,  6:32,  2 users,  load average: 2.15, 1.24, 1.04
Tasks: 180 total,   1 running, 179 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.6%us,  1.0%sy,  0.0%ni, 95.0%id,  1.7%wa,  0.0%hi,  0.7%si,  0.0%st
Mem:  66008460k total, 21226356k used, 44782104k free,   817252k buffers
Swap: 61437944k total,        0k used, 61437944k free, 17404540k cached
Change delay from 3.0 to: 
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                    
18042 root      20   0 15076 1272  904 S 100.0  0.0  5109402h top                                                       
18041 root      20   0  105m 1192 1028 S 100.0  0.0 900776:13 sh                                                        
18043 root      20   0 98.6m  808  692 S 100.0  0.0 900776:13 iostat                                                    
    1 root      20   0 19276 1508 1228 S  0.0  0.0   0:02.40 init                                                       
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kthreadd                                                   
    3 root      20   0     0    0    0 S  0.0  0.0  5119411h ksoftirqd/0                                                
    5 root      20   0     0    0    0 S  0.0  0.0 10218485h kworker/u:0                                                
    6 root      RT   0     0    0    0 S  0.0  0.0 986912:21 migration/0                                                
    7 root      RT   0     0    0    0 S  0.0  0.0 906076:59 migration/1                                                
    9 root      20   0     0    0    0 S  0.0  0.0  45038,51 ksoftirqd/1                                                
   11 root      RT   0     0    0    0 S  0.0  0.0 911444:25 migration/2                                                
   13 root      20   0     0    0    0 S  0.0  0.0 120103,33 ksoftirqd/2                                                
   14 root      RT   0     0    0    0 S  0.0  0.0 941393:43 migration/3

strace shows that these proceses hangs on select() system call

# strace -p 1938
Process 1938 attached - interrupt to quit
select(12, [0 11], NULL, NULL, NULL^C <unfinished ...>
Process 1938 detached

What can be the reason for this?

Best Answer

I struggled with a similar situation today. My friends were gdb, rb_backtrace() and rake debug. And in my case, it was an infinite while loop under some rare circumstances.

Just rb_backtrace() to print out the backtrace. In my case, it was printed out in the apache's error log (/var/log/apache2/error.log).

This backtrace is in reverse order than usual. See the last lines, and look for lines referring to your application code. Then, use a ruby debugger to debug just before that line.

This post https://robots.thoughtbot.com/using-gdb-to-inspect-a-running-ruby-process was helpful.

Related Topic