We upgraded to php7 (this was only change in server), this is causing high load on CPU.
Cacti graph
sar output
04:40:01 AM CPU %user %nice %system %iowait %steal %idle
04:50:01 AM all 2.85 0.00 2.13 0.01 0.00 95.01
04:50:01 AM 0 4.53 0.00 8.41 0.03 0.00 87.03
04:50:01 AM 1 24.64 0.00 3.93 0.01 0.00 71.43
04:50:01 AM 2 9.01 0.00 4.28 0.00 0.00 86.71
04:50:01 AM 3 3.89 0.00 2.23 0.00 0.00 93.87
04:50:01 AM 4 0.49 0.00 0.20 0.00 0.00 99.31
04:50:01 AM 5 0.38 0.00 0.79 0.00 0.00 98.83
04:50:01 AM 6 5.15 0.00 13.87 0.04 0.00 80.94
04:50:01 AM 7 1.18 0.00 5.05 0.00 0.00 93.76
04:50:01 AM 8 0.61 0.00 1.94 0.00 0.00 97.45
04:50:01 AM 9 0.37 0.00 1.33 0.00 0.00 98.30
04:50:01 AM 10 0.05 0.00 0.30 0.00 0.00 99.64
04:50:01 AM 11 0.11 0.00 0.03 0.00 0.00 99.85
04:50:01 AM 12 13.35 0.00 2.62 0.02 0.00 84.00
04:50:01 AM 13 0.98 0.00 0.24 0.00 0.00 98.78
04:50:01 AM 14 0.59 0.00 0.36 0.00 0.00 99.05
04:50:01 AM 15 0.24 0.00 0.13 0.00 0.00 99.63
04:50:01 AM 16 0.24 0.00 0.14 0.00 0.00 99.62
04:50:01 AM 17 0.03 0.00 0.03 0.00 0.00 99.94
04:50:01 AM 18 0.99 0.00 3.73 0.02 0.00 95.27
04:50:01 AM 19 0.37 0.00 0.54 0.00 0.00 99.09
04:50:01 AM 20 0.62 0.00 0.56 0.00 0.00 98.82
04:50:01 AM 21 0.24 0.00 0.28 0.00 0.00 99.48
04:50:01 AM 22 0.29 0.00 0.21 0.00 0.00 99.49
04:50:01 AM 23 0.07 0.00 0.07 0.00 0.00 99.87
04:50:01 AM CPU %user %nice %system %iowait %steal %idle
05:00:01 AM all 5.66 0.00 2.19 0.01 0.00 92.13
05:00:01 AM 0 7.73 0.00 7.75 0.05 0.00 84.47
05:00:01 AM 1 49.79 0.00 8.65 0.00 0.00 41.55
05:00:01 AM 2 25.74 0.00 5.77 0.00 0.00 68.50
05:00:01 AM 3 4.95 0.00 1.42 0.00 0.00 93.63
05:00:01 AM 4 1.89 0.00 1.69 0.00 0.00 96.42
05:00:01 AM 5 0.38 0.00 0.97 0.00 0.00 98.65
05:00:01 AM 6 9.79 0.00 3.96 0.10 0.00 86.15
05:00:01 AM 7 2.09 0.00 5.39 0.02 0.00 92.50
05:00:01 AM 8 0.31 0.00 0.24 0.01 0.00 99.44
05:00:01 AM 9 0.76 0.00 1.08 0.00 0.00 98.16
05:00:01 AM 10 0.02 0.00 0.13 0.01 0.00 99.84
05:00:01 AM 11 0.33 0.00 1.33 0.00 0.00 98.34
05:00:01 AM 12 25.21 0.00 3.81 0.02 0.00 70.96
05:00:01 AM 13 0.90 0.00 0.66 0.00 0.00 98.44
05:00:01 AM 14 0.67 0.00 0.76 0.05 0.00 98.52
05:00:01 AM 15 0.31 0.00 0.27 0.00 0.00 99.42
05:00:01 AM 16 0.28 0.00 0.26 0.00 0.00 99.46
05:00:01 AM 17 0.12 0.00 0.18 0.00 0.00 99.69
05:00:01 AM 18 1.24 0.00 5.03 0.04 0.00 93.69
05:00:01 AM 19 0.89 0.00 0.87 0.03 0.00 98.20
05:00:01 AM 20 1.74 0.00 1.23 0.00 0.00 97.02
05:00:01 AM 21 0.44 0.00 0.42 0.00 0.00 99.14
05:00:01 AM 22 0.38 0.00 0.45 0.00 0.00 99.17
05:00:01 AM 23 0.09 0.00 0.47 0.00 0.00 99.44
05:00:01 AM CPU %user %nice %system %iowait %steal %idle
05:10:01 AM all 6.16 0.00 1.64 0.02 0.00 92.18
05:10:01 AM 0 6.03 0.00 5.99 0.09 0.00 87.90
05:10:01 AM 1 48.29 0.00 7.89 0.00 0.00 43.81
05:10:01 AM 2 30.24 0.00 5.09 0.00 0.00 64.67
05:10:01 AM 3 9.02 0.00 1.20 0.00 0.00 89.77
05:10:01 AM 4 2.13 0.00 0.36 0.00 0.00 97.51
05:10:01 AM 5 0.54 0.00 0.16 0.00 0.00 99.30
05:10:01 AM 6 10.47 0.00 5.23 0.15 0.00 84.15
05:10:01 AM 7 1.90 0.00 1.13 0.02 0.00 96.95
05:10:01 AM 8 0.66 0.00 1.85 0.01 0.00 97.47
05:10:01 AM 9 0.26 0.00 0.13 0.00 0.00 99.61
05:10:01 AM 10 0.02 0.00 0.08 0.00 0.00 99.90
05:10:01 AM 11 0.01 0.00 0.08 0.00 0.00 99.92
05:10:01 AM 12 28.45 0.00 3.41 0.06 0.00 68.09
05:10:01 AM 13 1.60 0.00 0.73 0.00 0.00 97.67
05:10:01 AM 14 0.55 0.00 0.67 0.00 0.00 98.78
05:10:01 AM 15 0.66 0.00 0.24 0.00 0.00 99.09
05:10:01 AM 16 0.44 0.00 0.18 0.00 0.00 99.39
05:10:01 AM 17 0.19 0.00 0.12 0.00 0.00 99.69
05:10:01 AM 18 0.39 0.00 0.53 0.03 0.00 99.05
05:10:01 AM 19 2.51 0.00 1.42 0.00 0.00 96.07
05:10:01 AM 20 1.11 0.00 1.15 0.00 0.00 97.74
05:10:01 AM 21 1.99 0.00 1.25 0.00 0.00 96.76
05:10:01 AM 22 0.35 0.00 0.35 0.00 0.00 99.29
05:10:01 AM 23 0.22 0.00 0.34 0.01 0.00 99.44
05:10:01 AM CPU %user %nice %system %iowait %steal %idle
05:20:01 AM all 4.02 0.00 12.27 0.01 0.00 83.69
05:20:01 AM 0 7.35 0.00 18.25 0.02 0.00 74.38
05:20:01 AM 1 11.90 0.00 31.41 0.05 0.00 56.64
05:20:01 AM 2 6.57 0.00 25.74 0.00 0.00 67.68
05:20:01 AM 3 3.52 0.00 19.25 0.00 0.00 77.23
05:20:01 AM 4 2.58 0.00 16.29 0.00 0.00 81.14
05:20:01 AM 5 2.12 0.00 12.28 0.02 0.00 85.58
05:20:01 AM 6 12.51 0.00 25.54 0.06 0.00 61.88
05:20:01 AM 7 4.24 0.00 16.93 0.00 0.00 78.82
05:20:01 AM 8 3.75 0.00 8.85 0.01 0.00 87.40
05:20:01 AM 9 2.45 0.00 5.40 0.01 0.00 92.15
05:20:01 AM 10 2.79 0.00 4.57 0.00 0.00 92.64
05:20:01 AM 11 1.00 0.00 5.14 0.00 0.00 93.86
05:20:01 AM 12 5.14 0.00 19.32 0.02 0.00 75.52
05:20:01 AM 13 2.28 0.00 16.37 0.00 0.00 81.35
05:20:01 AM 14 1.26 0.00 11.01 0.00 0.00 87.73
05:20:01 AM 15 0.80 0.00 8.60 0.00 0.00 90.60
05:20:01 AM 16 0.93 0.00 5.01 0.00 0.00 94.06
05:20:01 AM 17 1.41 0.00 4.92 0.00 0.00 93.67
05:20:01 AM 18 6.21 0.00 12.08 0.04 0.00 81.67
05:20:01 AM 19 4.66 0.00 7.26 0.00 0.00 88.08
05:20:01 AM 20 0.70 0.00 4.83 0.00 0.00 94.47
05:20:01 AM 21 5.03 0.00 5.96 0.00 0.00 89.00
05:20:01 AM 22 7.01 0.00 5.47 0.00 0.00 87.52
05:20:01 AM 23 0.30 0.00 4.15 0.00 0.00 95.55
05:20:01 AM CPU %user %nice %system %iowait %steal %idle
05:30:01 AM all 4.34 0.00 11.07 0.01 0.00 84.58
05:30:01 AM 0 5.33 0.00 16.49 0.02 0.00 78.17
05:30:01 AM 1 17.19 0.00 33.85 0.03 0.00 48.93
05:30:01 AM 2 11.77 0.00 25.47 0.00 0.00 62.76
05:30:01 AM 3 5.08 0.00 21.68 0.00 0.00 73.23
05:30:01 AM 4 6.13 0.00 14.52 0.00 0.00 79.35
05:30:01 AM 5 2.06 0.00 11.81 0.00 0.00 86.13
05:30:01 AM 6 16.30 0.00 26.25 0.03 0.00 57.42
05:30:01 AM 7 3.80 0.00 17.13 0.01 0.00 79.06
05:30:01 AM 8 4.56 0.00 7.31 0.00 0.00 88.13
05:30:01 AM 9 0.11 0.00 2.77 0.01 0.00 97.11
05:30:01 AM 10 0.51 0.00 2.72 0.00 0.00 96.77
05:30:01 AM 11 4.69 0.00 2.10 0.00 0.00 93.21
05:30:01 AM 12 8.04 0.00 20.36 0.02 0.00 71.58
05:30:01 AM 13 1.30 0.00 12.20 0.00 0.00 86.50
05:30:01 AM 14 3.16 0.00 9.36 0.00 0.00 87.47
05:30:01 AM 15 0.51 0.00 4.93 0.01 0.00 94.55
05:30:01 AM 16 0.47 0.00 4.21 0.00 0.00 95.32
05:30:01 AM 17 0.43 0.00 3.02 0.00 0.00 96.55
05:30:01 AM 18 6.61 0.00 14.71 0.02 0.00 78.66
05:30:01 AM 19 3.60 0.00 6.18 0.00 0.00 90.22
05:30:01 AM 20 1.95 0.00 3.81 0.00 0.00 94.24
05:30:01 AM 21 0.46 0.00 2.55 0.00 0.00 96.99
05:30:01 AM 22 0.12 0.00 1.47 0.00 0.00 98.41
05:30:01 AM 23 0.13 0.00 0.88 0.00 0.00 98.99
05:30:01 AM CPU %user %nice %system %iowait %steal %idle
05:40:02 AM all 3.23 0.00 14.96 0.02 0.00 81.80
05:40:02 AM 0 5.70 0.00 21.45 0.18 0.00 72.67
05:40:02 AM 1 18.01 0.00 39.59 0.01 0.00 42.39
05:40:02 AM 2 9.73 0.00 33.71 0.01 0.00 56.55
05:40:02 AM 3 6.22 0.00 27.88 0.01 0.00 65.90
05:40:02 AM 4 2.83 0.00 20.78 0.00 0.00 76.39
05:40:02 AM 5 2.48 0.00 14.93 0.01 0.00 82.57
05:40:02 AM 6 5.30 0.00 35.07 0.05 0.00 59.57
05:40:02 AM 7 1.29 0.00 22.27 0.00 0.00 76.44
05:40:02 AM 8 0.77 0.00 9.50 0.00 0.00 89.73
05:40:02 AM 9 1.67 0.00 6.24 0.00 0.00 92.09
05:40:02 AM 10 0.88 0.00 4.54 0.00 0.00 94.59
05:40:02 AM 11 1.10 0.00 4.21 0.00 0.00 94.69
05:40:02 AM 12 9.25 0.00 26.51 0.01 0.00 64.23
05:40:02 AM 13 2.30 0.00 15.82 0.00 0.00 81.87
05:40:02 AM 14 1.31 0.00 11.75 0.00 0.00 86.94
05:40:02 AM 15 0.66 0.00 10.17 0.00 0.00 89.18
05:40:02 AM 16 1.17 0.00 5.83 0.00 0.00 93.01
05:40:02 AM 17 0.60 0.00 4.94 0.00 0.00 94.46
05:40:02 AM 18 2.62 0.00 20.10 0.12 0.00 77.16
05:40:02 AM 19 0.83 0.00 9.10 0.00 0.00 90.08
05:40:02 AM 20 0.75 0.00 5.42 0.00 0.00 93.82
05:40:02 AM 21 1.04 0.00 3.14 0.00 0.00 95.82
05:40:02 AM 22 0.28 0.00 2.73 0.00 0.00 96.99
05:40:02 AM 23 0.67 0.00 3.34 0.00 0.00 95.99
05:40:02 AM CPU %user %nice %system %iowait %steal %idle
05:50:01 AM all 2.28 0.00 11.49 0.04 0.00 86.19
05:50:01 AM 0 3.58 0.00 16.41 0.03 0.00 79.98
05:50:01 AM 1 14.45 0.00 32.73 0.01 0.00 52.82
05:50:01 AM 2 7.32 0.00 25.51 0.02 0.00 67.15
05:50:01 AM 3 3.46 0.00 20.23 0.00 0.00 76.31
05:50:01 AM 4 2.44 0.00 12.95 0.01 0.00 84.60
05:50:01 AM 5 1.33 0.00 12.12 0.00 0.00 86.55
05:50:01 AM 6 4.11 0.00 29.63 0.46 0.00 65.80
05:50:01 AM 7 1.67 0.00 14.28 0.06 0.00 83.99
05:50:01 AM 8 0.58 0.00 4.48 0.01 0.00 94.92
05:50:01 AM 9 0.24 0.00 4.33 0.02 0.00 95.41
05:50:01 AM 10 0.07 0.00 1.87 0.00 0.00 98.05
05:50:01 AM 11 0.08 0.00 1.36 0.00 0.00 98.55
05:50:01 AM 12 7.46 0.00 19.27 0.01 0.00 73.27
05:50:01 AM 13 1.08 0.00 13.72 0.00 0.00 85.20
05:50:01 AM 14 0.71 0.00 10.76 0.02 0.00 88.51
05:50:01 AM 15 0.66 0.00 7.16 0.01 0.00 92.17
05:50:01 AM 16 0.36 0.00 4.58 0.00 0.00 95.06
05:50:01 AM 17 0.49 0.00 3.64 0.00 0.00 95.87
05:50:01 AM 18 0.90 0.00 19.63 0.11 0.00 79.36
05:50:01 AM 19 0.76 0.00 6.19 0.03 0.00 93.02
05:50:01 AM 20 1.11 0.00 4.42 0.06 0.00 94.41
05:50:01 AM 21 0.74 0.00 3.29 0.02 0.00 95.95
05:50:01 AM 22 0.37 0.00 1.72 0.01 0.00 97.90
05:50:01 AM 23 0.78 0.00 5.67 0.04 0.00 93.50
05:50:01 AM CPU %user %nice %system %iowait %steal %idle
06:00:01 AM all 1.93 0.00 11.03 0.01 0.00 87.03
06:00:01 AM 0 3.39 0.00 17.41 0.02 0.00 79.18
06:00:01 AM 1 14.49 0.00 35.42 0.00 0.00 50.09
06:00:01 AM 2 5.96 0.00 27.15 0.00 0.00 66.89
06:00:01 AM 3 3.31 0.00 18.51 0.02 0.00 78.16
06:00:01 AM 4 1.45 0.00 11.41 0.00 0.00 87.15
06:00:01 AM 5 0.50 0.00 7.73 0.00 0.00 91.77
06:00:01 AM 6 2.20 0.00 35.46 0.04 0.00 62.30
06:00:01 AM 7 1.59 0.00 15.17 0.01 0.00 83.24
06:00:01 AM 8 0.23 0.00 3.56 0.01 0.00 96.21
06:00:01 AM 9 0.14 0.00 2.01 0.00 0.00 97.85
06:00:01 AM 10 0.13 0.00 1.37 0.00 0.00 98.51
06:00:01 AM 11 0.04 0.00 0.73 0.00 0.00 99.23
06:00:01 AM 12 7.45 0.00 22.89 0.01 0.00 69.66
06:00:01 AM 13 1.45 0.00 13.04 0.00 0.00 85.51
06:00:01 AM 14 0.82 0.00 8.06 0.00 0.00 91.12
06:00:01 AM 15 0.31 0.00 4.05 0.00 0.00 95.63
06:00:01 AM 16 0.29 0.00 2.70 0.00 0.00 97.02
06:00:01 AM 17 0.13 0.00 0.97 0.00 0.00 98.90
06:00:01 AM 18 1.15 0.00 21.66 0.08 0.00 77.10
06:00:01 AM 19 0.43 0.00 7.24 0.00 0.00 92.33
06:00:01 AM 20 0.39 0.00 4.84 0.00 0.00 94.77
06:00:01 AM 21 0.25 0.00 1.94 0.00 0.00 97.81
06:00:01 AM 22 0.15 0.00 0.64 0.00 0.00 99.21
06:00:01 AM 23 0.13 0.00 0.91 0.00 0.00 98.96
06:00:01 AM CPU %user %nice %system %iowait %steal %idle
06:10:01 AM all 2.78 0.00 18.12 0.01 0.00 79.09
06:10:01 AM 0 4.13 0.00 25.91 0.07 0.00 69.89
06:10:01 AM 1 14.89 0.00 41.23 0.01 0.00 43.88
06:10:01 AM 2 8.86 0.00 35.86 0.01 0.00 55.28
06:10:01 AM 3 4.97 0.00 31.55 0.00 0.00 63.48
06:10:01 AM 4 2.86 0.00 25.14 0.00 0.00 72.00
06:10:01 AM 5 2.20 0.00 19.33 0.00 0.00 78.48
06:10:01 AM 6 5.45 0.00 36.19 0.02 0.00 58.34
06:10:01 AM 7 1.96 0.00 26.27 0.04 0.00 71.72
06:10:01 AM 8 1.26 0.00 12.71 0.03 0.00 86.01
06:10:01 AM 9 0.67 0.00 9.15 0.01 0.00 90.18
06:10:01 AM 10 0.55 0.00 5.23 0.00 0.00 94.22
06:10:01 AM 11 0.74 0.00 5.65 0.00 0.00 93.61
06:10:01 AM 12 6.99 0.00 28.78 0.01 0.00 64.23
06:10:01 AM 13 1.61 0.00 19.93 0.00 0.00 78.46
06:10:01 AM 14 1.53 0.00 16.81 0.00 0.00 81.66
06:10:01 AM 15 1.30 0.00 13.27 0.00 0.00 85.43
06:10:01 AM 16 1.07 0.00 10.22 0.00 0.00 88.71
06:10:01 AM 17 0.93 0.00 8.63 0.00 0.00 90.44
06:10:01 AM 18 0.93 0.00 24.76 0.03 0.00 74.28
06:10:01 AM 19 0.90 0.00 13.35 0.01 0.00 85.74
06:10:01 AM 20 0.85 0.00 9.04 0.00 0.00 90.11
06:10:01 AM 21 1.00 0.00 6.38 0.02 0.00 92.60
06:10:01 AM 22 0.54 0.00 6.25 0.00 0.00 93.21
06:10:01 AM 23 0.46 0.00 3.31 0.01 0.00 96.23
If you notice, System time increased after 5:20 am.
Strace output of php process
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
41.96 0.001331 3 438 poll
30.14 0.000956 2 548 munmap
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
52.21 0.001264 6 222 munmap
11.61 0.000281 1 247 poll
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
44.56 0.001159 5 225 munmap
37.99 0.000988 3 320 poll
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.000539 67 8 munmap
0.00 0.000000 0 119 write
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
51.37 0.001034 6 161 munmap
21.56 0.000434 1 293 poll
How to troubleshoot high load caused by php7
UPDATE 1
- This problem does not occurs when total RAM on system is low (<32GB)
but it is occurring on system with high RAM (256GB). - Restarting PHP sometimes fixes this problem for few days.
Best Answer
All of the time appears to be going on munmap. Which is unmapping memory. PHP7 has the capability to use Transparent Huge Pages in the Kernel.
Transparent huge pages have caused issues in the past with different kernels and software. https://answers.splunk.com/answers/188875/how-do-i-disable-transparent-huge-pages-thp-and-co.html shows how this can be an issue and how to turn it off.
I have had problems with Huge Pages and PostgreSQL on RHEL 6 using this kernel. Turning off huge pages resolved the issues.
You have also submitted https://bugs.php.net/bug.php?id=72322 directly to the PHP project to get answers there as well.