I am having a heck of a time figuring this out. Apache, every few days hangs for about 20 or so minutes and then "comes back to life". This has happened during the middle of the day, it's happened in the middle of the night. The server is a beefy web server with 4 CPUs 8GB RAM and another 12GB swap. I cannot see anything that stands out in the error logs. Can someone see anything in these logs to indicate the PROBLEM? As everything I see looks like a result of the problem or a symptom!
syslog
Dec 10 05:25:02 admin kernel: [397885.197196] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23622 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:03 admin kernel: [397886.194931] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23623 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:05 admin kernel: [397888.198941] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23624 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:08 admin kernel: [397891.641472] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=967 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:08 admin kernel: [397891.974137] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28012 DF PROTO=TCP SPT=50916 DPT=443 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:08 admin kernel: [397891.974230] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=18.206.39.189 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=116 ID=28013 DF PROTO=TCP SPT=50866 DPT=80 WINDOW=1021 RES=0x00 ACK FIN URGP=0
Dec 10 05:25:09 admin kernel: [397892.210857] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23625 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:09 admin CRON[8325]: (root) CMD (sh /etc/apache2/websitesCron/apacheTest.sh)
Dec 10 05:25:09 admin CRON[8326]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 388
Dec 10 05:25:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 388
Dec 10 05:25:17 admin kernel: [397900.226738] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23626 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 401
Dec 10 05:25:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 401
Dec 10 05:25:27 admin kernel: [397910.285120] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=985 DF PROTO=TCP SPT=52268 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
Dec 10 05:25:33 admin kernel: [397916.242478] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23627 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: from=root, size=425, class=0, nrcpts=1, msgid=<[email protected]>, bodytype=8BITMIME, relay=root@localhost
Dec 10 05:25:34 admin sendmail[8390]: 1BABPXbK008390: to=root, delay=00:00:01, mailer=relay, pri=30425, stat=queued
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 413
Dec 10 05:25:40 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 413
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 419
Dec 10 05:25:55 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 419
Dec 10 05:26:04 admin CRON[8459]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 05:26:05 admin kernel: [397948.305893] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:50:56:08:09:19:08:00 SRC=10.2.6.60 DST=10.2.6.80 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23628 DF PROTO=TCP SPT=59284 DPT=4949 WINDOW=29200 RES=0x00 SYN URGP=0
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 423
Dec 10 05:26:10 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 423
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 428
Dec 10 05:26:25 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 428
Dec 10 05:26:26 admin kernel: [397969.985342] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=990 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:29 admin kernel: [397972.391234] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=995 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:34 admin kernel: [397977.203821] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=20.115.4.12 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=115 ID=999 DF PROTO=TCP SPT=62843 DPT=80 WINDOW=2045 RES=0x00 ACK FIN URGP=0
Dec 10 05:26:40 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 436
Dec 10 05:26:40 admin kernel: [397983.346629] apache2 invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Dec 10 05:26:40 admin kernel: [397983.346635] apache2 cpuset=/ mems_allowed=0
Dec 10 05:26:40 admin kernel: [397983.346644] CPU: 0 PID: 8270 Comm: apache2 Not tainted 4.4.0-119-generic #143-Ubuntu
Dec 10 05:26:40 admin kernel: [397983.346646] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Dec 10 05:26:40 admin kernel: [397983.346649] 0000000000000286 553e9d17649eeffd ffff8800138a7b10 ffffffff81400443
Dec 10 05:26:40 admin kernel: [397983.346652] ffff8800138a7cc8 ffff8800b8e5aa00 ffff8800138a7b80 ffffffff8121086e
Dec 10 05:26:40 admin kernel: [397983.346655] ffff88023fc1ad70 ffff88023fc1ad60 ffffea0006a80b00 0000000100000001
Dec 10 05:26:40 admin kernel: [397983.346657] Call Trace:
Dec 10 05:26:40 admin kernel: [397983.346670] [<ffffffff81400443>] dump_stack+0x63/0x90
Dec 10 05:26:40 admin kernel: [397983.346677] [<ffffffff8121086e>] dump_header+0x5a/0x1c5
Dec 10 05:26:40 admin kernel: [397983.346682] [<ffffffff81196f32>] oom_kill_process+0x202/0x3c0
Dec 10 05:26:40 admin kernel: [397983.346685] [<ffffffff81197359>] out_of_memory+0x219/0x460
Dec 10 05:26:40 admin kernel: [397983.346689] [<ffffffff8119d3a5>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
Dec 10 05:26:40 admin kernel: [397983.346692] [<ffffffff8119d7c8>] __alloc_pages_nodemask+0x288/0x2a0
Dec 10 05:26:40 admin kernel: [397983.346694] [<ffffffff8119d87b>] alloc_kmem_pages_node+0x4b/0xc0
Dec 10 05:26:40 admin kernel: [397983.346699] [<ffffffff8108077e>] copy_process+0x1be/0x1bb0
Dec 10 05:26:40 admin kernel: [397983.346703] [<ffffffff811a44f7>] ? lru_cache_add_active_or_unevictable+0x27/0xa0
Dec 10 05:26:40 admin kernel: [397983.346707] [<ffffffff811c6178>] ? handle_mm_fault+0xcc8/0x1820
Dec 10 05:26:40 admin kernel: [397983.346709] [<ffffffff81082300>] _do_fork+0x80/0x360
Dec 10 05:26:40 admin kernel: [397983.346712] [<ffffffff81082689>] SyS_clone+0x19/0x20
Dec 10 05:26:40 admin kernel: [397983.346717] [<ffffffff8184f708>] entry_SYSCALL_64_fastpath+0x1c/0xbb
Dec 10 05:26:40 admin kernel: [397983.346718] Mem-Info:
Dec 10 05:26:40 admin kernel: [397983.346723] active_anon:1463690 inactive_anon:290454 isolated_anon:132
Dec 10 05:26:40 admin kernel: [397983.346723] active_file:17208 inactive_file:13369 isolated_file:0
Dec 10 05:26:40 admin kernel: [397983.346723] unevictable:913 dirty:0 writeback:987 unstable:0
Dec 10 05:26:40 admin kernel: [397983.346723] slab_reclaimable:19387 slab_unreclaimable:41949
Dec 10 05:26:40 admin kernel: [397983.346723] mapped:25866 shmem:18736 pagetables:141112 bounce:0
Dec 10 05:26:40 admin kernel: [397983.346723] free:26269 free_pcp:0 free_cma:0
Dec 10 05:26:40 admin kernel: [397983.346728] Node 0 DMA free:15864kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 10 05:26:40 admin kernel: [397983.346734] lowmem_reserve[]: 0 2937 7928 7928 7928
Dec 10 05:26:40 admin kernel: [397983.346737] Node 0 DMA32 free:44712kB min:24988kB low:31232kB high:37480kB active_anon:2037256kB inactive_anon:515680kB active_file:24624kB inactive_file:19936kB unevictable:1228kB isolated(anon):348kB isolated(file):0kB present:3129216kB managed:3048436kB mlocked:1228kB dirty:0kB writeback:2620kB mapped:38304kB shmem:28716kB slab_reclaimable:25488kB slab_unreclaimable:70356kB kernel_stack:15136kB pagetables:261600kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346743] lowmem_reserve[]: 0 0 4990 4990 4990
Dec 10 05:26:40 admin kernel: [397983.346745] Node 0 Normal free:44500kB min:42460kB low:53072kB high:63688kB active_anon:3817504kB inactive_anon:646136kB active_file:44208kB inactive_file:33540kB unevictable:2424kB isolated(anon):180kB isolated(file):0kB present:5242880kB managed:5110492kB mlocked:2424kB dirty:0kB writeback:1328kB mapped:65160kB shmem:46228kB slab_reclaimable:52060kB slab_unreclaimable:97432kB kernel_stack:13728kB pagetables:302848kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 10 05:26:40 admin kernel: [397983.346750] lowmem_reserve[]: 0 0 0 0 0
Dec 10 05:26:40 admin kernel: [397983.346752] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15864kB
Dec 10 05:26:40 admin kernel: [397983.346762] Node 0 DMA32: 6731*4kB (UM) 2257*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Dec 10 05:26:40 admin kernel: [397983.346768] Node 0 Normal: 10918*4kB (UMEH) 49*8kB (UMH) 0*16kB 1*32kB (H) 1*64kB (H) 3*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 45056kB
Dec 10 05:26:40 admin kernel: [397983.346777] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 10 05:26:40 admin kernel: [397983.346779] 68538 total pagecache pages
Dec 10 05:26:40 admin kernel: [397983.346781] 18641 pages in swap cache
Dec 10 05:26:40 admin kernel: [397983.346783] Swap cache stats: add 383468753, delete 383450112, find 229519152/378197444
Dec 10 05:26:40 admin kernel: [397983.346784] Free swap = 1252280kB
Dec 10 05:26:42 admin kernel: [397983.346785] Total swap = 11717628kB
Dec 10 05:26:42 admin kernel: [397983.346787] 2097022 pages RAM
Dec 10 05:26:42 admin kernel: [397983.346788] 0 pages HighMem/MovableOnly
Dec 10 05:26:42 admin kernel: [397983.346789] 53314 pages reserved
Dec 10 05:26:42 admin kernel: [397983.346790] 0 pages cma reserved
Dec 10 05:26:42 admin kernel: [397983.346791] 0 pages hwpoisoned
Dec 10 05:26:42 admin kernel: [397983.346793] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 10 05:26:42 admin kernel: [397983.346800] [ 409] 0 409 10976 822 24 3 602 0 systemd-journal
Dec 10 05:26:42 admin kernel: [397983.346803] [ 451] 0 451 23693 83 17 3 50 0 lvmetad
Dec 10 05:26:42 admin kernel: [397983.346805] [ 470] 0 470 11415 242 22 3 485 -1000 systemd-udevd
Dec 10 05:26:42 admin kernel: [397983.346808] [ 784] 0 784 47527 409 52 3 223 0 vmtoolsd
Dec 10 05:26:42 admin kernel: [397983.346811] [ 838] 0 838 5884 0 16 3 51 0 rpc.idmapd
Dec 10 05:26:42 admin kernel: [397983.346813] [ 842] 100 842 25081 132 20 3 52 0 systemd-timesyn
Dec 10 05:26:42 admin kernel: [397983.346816] [ 934] 0 934 11906 140 27 3 109 0 rpcbind
Dec 10 05:26:42 admin kernel: [397983.346818] [ 948] 0 948 151028 166 28 4 213 0 lxcfs
Dec 10 05:26:42 admin kernel: [397983.346821] [ 953] 0 953 7252 456 19 3 48 0 cron
Dec 10 05:26:42 admin kernel: [397983.346823] [ 959] 0 959 1099 335 8 3 40 0 acpid
Dec 10 05:26:42 admin kernel: [397983.346826] [ 961] 0 961 7165 239 19 3 57 0 systemd-logind
Dec 10 05:26:42 admin kernel: [397983.346828] [ 967] 108 967 10725 566 25 3 76 -900 dbus-daemon
Dec 10 05:26:42 admin kernel: [397983.346831] [ 1047] 0 1047 21359 342 32 3 347 0 VGAuthService
Dec 10 05:26:42 admin kernel: [397983.346833] [ 1049] 0 1049 68974 456 37 3 224 0 accounts-daemon
Dec 10 05:26:42 admin kernel: [397983.346835] [ 1051] 0 1051 6511 391 18 3 47 0 atd
Dec 10 05:26:42 admin kernel: [397983.346838] [ 1056] 104 1056 64099 457 29 3 367 0 rsyslogd
Dec 10 05:26:42 admin kernel: [397983.346841] [ 1148] 0 1148 3343 51 11 3 23 0 mdadm
Dec 10 05:26:42 admin kernel: [397983.346843] [ 1179] 0 1179 6011 251 16 3 89 0 vsftpd
Dec 10 05:26:42 admin kernel: [397983.346845] [ 1189] 0 1189 69278 506 40 4 122 0 polkitd
Dec 10 05:26:42 admin kernel: [397983.346848] [ 1194] 0 1194 1305 358 9 3 61 0 iscsid
Dec 10 05:26:42 admin kernel: [397983.346850] [ 1195] 0 1195 1430 879 10 3 0 -17 iscsid
Dec 10 05:26:42 admin kernel: [397983.346853] [ 1207] 0 1207 9494 0 22 3 190 0 rpc.mountd
Dec 10 05:26:42 admin kernel: [397983.346856] [ 1221] 0 1221 16378 339 36 3 196 -1000 sshd
Dec 10 05:26:42 admin kernel: [397983.346859] [ 1296] 0 1296 16458 453 37 4 128 0 login
Dec 10 05:26:42 admin kernel: [397983.346861] [ 1326] 0 1326 4905 284 14 3 39 0 irqbalance
Dec 10 05:26:42 admin kernel: [397983.346864] [ 1420] 0 1420 13514 229 30 3 2438 0 munin-node
Dec 10 05:26:42 admin kernel: [397983.346866] [ 1501] 0 1501 26199 440 51 4 470 0 sendmail-mta
Dec 10 05:26:42 admin kernel: [397983.346869] [ 1681] 0 1681 109282 257 171 3 1936 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346871] [ 1687] 33 1687 109282 184 152 3 1961 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346873] [ 1688] 33 1688 109282 184 152 3 1961 0 php-fpm7.0
Dec 10 05:26:42 admin kernel: [397983.346876] [16727] 1000 16727 11330 422 25 3 217 0 systemd
Dec 10 05:26:42 admin kernel: [397983.346878] [16731] 1000 16731 52186 0 37 3 501 0 (sd-pam)
Dec 10 05:26:42 admin kernel: [397983.346881] [16734] 1000 16734 5900 357 18 4 725 0 bash
Dec 10 05:26:42 admin kernel: [397983.346884] [11281] 119 11281 69349 594 88 3 1257 0 freshclam
Dec 10 05:26:42 admin kernel: [397983.346886] [32663] 0 32663 23229 459 48 3 258 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346889] [32740] 1000 32740 23668 373 48 3 743 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346892] [22943] 0 22943 23229 330 51 3 272 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346894] [23020] 1000 23020 23229 332 49 3 247 0 sshd
Dec 10 05:26:42 admin kernel: [397983.346897] [23023] 1000 23023 3220 393 12 3 51 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346899] [24027] 1000 24027 3220 381 12 3 67 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346902] [27215] 1000 27215 3220 378 12 3 95 0 sftp-server
Dec 10 05:26:42 admin kernel: [397983.346904] [13006] 0 13006 168834 2419 280 3 29858 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346907] [15772] 33 15772 385874 34986 583 4 67131 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346909] [15773] 33 15773 381008 40856 579 4 53799 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346912] [15776] 33 15776 297940 35288 395 4 53305 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346915] [15777] 33 15777 303389 37051 415 4 49856 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346919] [15779] 33 15779 303184 39806 403 4 46606 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346922] [15782] 33 15782 304265 46518 404 4 43859 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346925] [15786] 33 15786 371258 41084 557 4 49806 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346929] [15787] 33 15787 302813 34962 403 4 53776 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346931] [15789] 33 15789 380748 35019 567 4 52932 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346934] [15799] 33 15799 384260 39566 561 4 55415 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346936] [15800] 33 15800 374314 29895 544 4 52106 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346939] [15801] 33 15801 380377 34701 565 4 53467 0 apache2
Dec 10 05:26:42 admin kernel: [397983.346941] [15802] 33 15802 306345 35364 424 4 57003 0 apache2
........
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 570
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 556
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 539
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 525
Dec 10 05:33:29 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 525
......
Dec 10 06:27:15 admin kernel: [401618.309140] Out of memory: Kill process 15801 (apache2) score 14 or sacrifice child
Dec 10 06:27:15 admin kernel: [401618.309960] Killed process 15801 (apache2) total-vm:1521508kB, anon-rss:23236kB, file-rss:0kB
Dec 10 06:27:21 admin CRON[12717]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 459
Dec 10 06:27:21 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 459
Dec 10 06:27:29 admin kernel: [401632.351612] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=209.85.238.216 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=42747 PROTO=TCP SPT=49490 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 475
Dec 10 06:27:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 475
Dec 10 06:27:43 admin apache2[12635]: *
Dec 10 06:27:43 admin systemd[1]: Stopped LSB: Apache2 web server.
Dec 10 06:27:43 admin systemd[1]: Starting LSB: Apache2 web server...
Dec 10 06:27:43 admin apache2[12749]: * Starting Apache httpd web server apache2
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 370
Dec 10 06:27:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 370
Dec 10 06:27:49 admin systemd[1]: Starting Daily apt upgrade and clean activities...
Dec 10 06:27:49 admin kernel: [401652.463319] [UFW BLOCK] IN=ens32 OUT=
MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=123.183.224.66 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=41274 PROTO=TCP SPT=45696 DPT=443 WINDOW=243 RES=0x00 ACK RST URGP=0
Dec 10 06:27:54 admin apache2[12749]: [Fri Dec 10 06:27:54.747509 2021] [proxy_html:notice] [pid 12805] AH01425: I18n support in mod_proxy_html requires mod_xml2enc. Without it, non-ASCII characters in proxied pages are likely to display incorrectly.
c 10 06:28:09 admin kernel: [401672.878972] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=76.99.197.116 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=52726 PROTO=TCP SPT=55476 DPT=443 WINDOW=248 RES=0x00 ACK RST URGP=0
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 225
Dec 10 06:28:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 225
Dec 10 06:28:30 admin kernel: [401693.571187] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=185.191.171.2 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=57 ID=0 DF PROTO=TCP SPT=42580 DPT=443 WINDOW=0 RES=0x00 RST URGP=0
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 176
Dec 10 06:28:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 176
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 139
Dec 10 06:28:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 139
Dec 10 06:28:49 admin kernel: [401712.942365] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=95.217.225.110 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=44271 PROTO=TCP SPT=36124 DPT=443 WINDOW=254 RES=0x00 ACK RST URGP=0
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 109
Dec 10 06:29:00 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 109
Dec 10 06:29:01 admin CRON[13228]: (root) CMD (cp /var/spool/cron/crontabs/root /var/www/crontab/root && chmod 777 /var/www/crontab/root && chown zak:zak /var/www/crontab/root)
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 87
Dec 10 06:29:15 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 87
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 69
Dec 10 06:29:30 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 69
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MTA-v4: load average: 55
Dec 10 06:29:45 admin sm-mta[1501]: rejecting connections on daemon MSP-v4: load average: 55
Dec 10 06:29:53 admin kernel: [401776.578209] [UFW BLOCK] IN=ens32 OUT= MAC=00:50:56:08:0d:03:00:a0:c9:27:01:01:08:00 SRC=116.179.37.124 DST=10.2.6.80 LEN=40 TOS=0x00 PREC=0x00 TTL=255 ID=47773 PROTO=TCP SPT=52871 DPT=443 WINDOW=246 RES=0x00 ACK RST URGP=0
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MTA-v4
Dec 10 06:30:00 admin sm-mta[1501]: accepting connections again for daemon MSP-v4
Apache error log
[Fri Dec 10 05:23:33.884301 2021] [pagespeed:warn] [pid 7829] [mod_pagespeed 1.13.35.2-0 @7829] Fetch timed out: https://|-REMOVED-|/css/fontawesome.css (connecting to:10.2.6.80) (4) waiting for 50 ms
[Fri Dec 10 05:23:33.927235 2021] [pagespeed:error] [pid 7881] [mod_pagespeed 1.13.35.2-0 @7881] Slow write operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/animate.css+stylesheet-1625614552.css+Contact-Us-1625614552.css.pagespeed.cc.0HlzBptJqh.css,.tempC6gpkp: 2974.19ms; configure SlowFileLatencyUs to change threshold\n
[Fri Dec 10 05:23:34.012843 2021] [pagespeed:warn] [pid 15909] [mod_pagespeed 1.13.35.2-0 @15909] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/buttons.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.036279 2021] [pagespeed:warn] [pid 7368] [mod_pagespeed 1.13.35.2-0 @7368] Fetch timed out: https://|-REMOVED-|/css/images/layers-2x.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.190221 2021] [pagespeed:warn] [pid 7615] [mod_pagespeed 1.13.35.2-0 @7615] Fetch timed out: https://|-REMOVED-|/imageserver/confirm/ie.png (connecting to:10.2.6.80) (1) waiting for 50 ms
[Fri Dec 10 05:23:34.291732 2021] [pagespeed:error] [pid 7314] [mod_pagespeed 1.13.35.2-0 @7314] Slow ReadFile operation on file /var/cache/mod_pagespeed/v3/|-REMOVED-|/https,3A/,2F|-REMOVED-|/css/presidential-solaris-1.css,: 333.858ms; configure SlowFileLatencyUs to change threshold\n
.........
[Fri Dec 10 06:26:59.176654 2021] [core:warn] [pid 13006] AH00045: child process 7180 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619326 2021] [core:warn] [pid 13006] AH00045: child process 7181 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619410 2021] [core:warn] [pid 13006] AH00045: child process 7182 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619449 2021] [core:warn] [pid 13006] AH00045: child process 7183 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619492 2021] [core:warn] [pid 13006] AH00045: child process 7184 still did not exit, sending a SIGTERM
[Fri Dec 10 06:26:59.619529 2021] [core:warn] [pid 13006] AH00045: child process 7185 still did not exit, sending a SIGTERM
.......
[Fri Dec 10 06:27:23.205977 2021] [core:error] [pid 13006] AH00047: could not make child process 7184 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206200 2021] [core:error] [pid 13006] AH00047: could not make child process 7326 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206283 2021] [core:error] [pid 13006] AH00047: could not make child process 7338 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206418 2021] [core:error] [pid 13006] AH00047: could not make child process 7353 exit, attempting to continue anyway
[Fri Dec 10 06:27:23.206562 2021] [core:error] [pid 13006] AH00047: could not make child process 7362 exit, attempting to continue anyway
.........
Fri Dec 10 06:27:23.323233 2021] [mpm_prefork:notice] [pid 13006] AH00169: caught SIGTERM, shutting down
Here is what the downtime "looks" like:
And the exact times the our Downtime detector caught, were:
DOWN: 2021-12-10 05:26:33 UTC-6
UP: 2021-12-10 05:45:02 UTC-6
Best Answer
Well for anyone in my situation that experiences this in the future .. This is a solution, albeit a hackish one.
I noticed that if I turned PageSpeed off, the problem was limited. I looked at my average Apache request and it was 62MB. I divided that by the leftover RAM for running all other resources .. I had 6GB left. I changed Max Server and Max Client to 128 from 256 -- That helped -- But did not "solve" it.
I then noticed that pagespeed wasn't storing physical files in /var/cache/mod_pagespeed as quickly as I expected .. It is using RAM space instead, even though using the line inside of the
conffile:Then I had an idea .. What does pagespeed do during a graceful apache restart when RAM in non accessible .. It stores in the local
/var/cache/directory is what.So because my web configuration is static, and the config files never change .. I wrote
service apache reloadinto a cron that runs every 30 mins. Now Pagespeed fills the disk invar/cacheand does not fill the RAM until implosion.NOTE If you are running 1000+ websites like we are .. Make sure you have ample room to cache every site's physical files .. This will end up being GBs worth of space .. But it's how PageSpeed works. Otherwise you'll fill your disk up in a hurry. My disk usage went from 102GB to 165GB usage within hours .. On a 1TB drive, that's trivial .. But if I had a 200GB drive it might be concerning ..