Linux – top command – cpu from processes do not add up

cpu-usagelinuxtop

I understand the various types of cpu usage reported by the top command (6.5%us, 17.2%sy, 0.0%ni, etc…), but why does the total %CPU for each process not add up to any of the Cpu(s) values? For example, below the java process is consuming 77.5% CPU, yet Cpu(s) says 76.0% is still idle. Why is this? This is on a single core system.

top - 05:53:27 up 32 min,  2 users,  load average: 0.16, 0.29, 0.34
Tasks:  71 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
Cpu(s):  6.5%us, 17.2%sy,  0.0%ni, 76.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   1758616k total,   643432k used,  1115184k free,    12224k buffers
Swap:   917500k total,        0k used,   917500k free,   304608k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
 1307 tomcat    20   0  683m 287m 9528 S 77.5 16.8  10:54.99 java                                                                                                                                                                          
 1571 ec2-user  20   0  2592 1096  872 R  1.0  0.1   0:08.61 top                                                                                                                                                                           
    1 root      20   0  2892 1364 1168 S  0.0  0.1   0:00.28 init                                                                                                                                                                          
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

Edit: Turns out this is a dual core system. Here is updated output after pressing 1 while the top command is up:

top - 06:10:21 up 49 min,  2 users,  load average: 0.28, 0.37, 0.34
Tasks:  71 total,   1 running,  70 sleeping,   0 stopped,   0 zombie
Cpu0  :  9.9%us, 19.7%sy,  0.0%ni, 69.0%id,  0.0%wa,  0.0%hi,  1.4%si,  0.0%st
Cpu1  :  5.0%us, 10.0%sy,  0.0%ni, 85.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1758616k total,   677548k used,  1081068k free,    13296k buffers
Swap:   917500k total,        0k used,   917500k free,   305732k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                      
 1307 tomcat    20   0  683m 318m 9528 S 68.0 18.6  17:23.53 java                                                                                                                                                                          
    1 root      20   0  2892 1364 1168 S  0.0  0.1   0:00.28 init                                                                                                                                                                          
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                                                                                                      
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.91 ksoftirqd/0                                                                                                                                                                   
    4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

Best Answer

Those two information that you are comparing won't match, simply, because they are collected from different files. That is although top shows the information in the same terminal, they aren't collected from the same source.

I simply ran an strace on top (running in batch mode). This is where it shows the system wide CPU information.

16:04:04.081092 open("/proc/stat", O_RDONLY) = 6 <0.000022>
16:04:04.081154 lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0 <0.000015>
16:04:04.081211 lstat("/proc/stat", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0     <0.000013>
16:04:04.081267 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000013>
16:04:04.081334 fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000013>
16:04:04.081385 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f699ace2000 <0.000016>
16:04:04.081440 lseek(6, 0, SEEK_SET)   = 0 <0.000013>
16:04:04.081494 read(6, "cpu  302573 6910 83103 10092403 "..., 1024) = 1024 <0.000070>
16:04:04.081656 write(1, "%Cpu(s):  2.9 us,  0.8 sy,  0.1 "..., 80) = 80 <0.000034>
16:04:04.081763 write(1, "KiB Mem:   8048484 total,  41402"..., 73) = 73 <0.000035>
16:04:04.081858 write(1, "KiB Swap:  8060924 total,       "..., 72) = 72 <0.000034>
16:04:04.081940 write(1, "\n", 1)       = 1 <0.000026>

Now, if you see /proc/stat, it shows all the CPUs of the system. top knows that too, because before opening /proc/stat, it opens sys filesystem.

16:04:03.367339 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3 <0.000027>
16:04:03.367408 read(3, "0-7\n", 8192)  = 4 <0.000019>
16:04:03.367464 close(3)                = 0 <0.000015>

Now, when it comes to collecting individual process information, it gets that from /proc/pid/statm and /proc/pid/stat file. (replace pid with well, actual pid).

As you can see, /proc/stat is system wide information for ALL the CPUs and individual proc files for the pids are their specific pid-only information.

So, they won't match.

Related Topic