For a few months now I've been experiencing random crashes on my 2GB RAM VPS Linode server. Someone convinced me to switch to self-managed hosting and I for some reason thought I could handle it despite having no experience in the field. I'm going to provide as much information as possible to help:
Site in question is a WordPress/Woocommerce site, it does have quite a lot of plugins so I'm aware that's not ideal but they are mostly required. Concurrent users is usually around 10, doesn't really go over 20. I've noticed that I do tend to get a lot of random bots coming to site that sometimes tends to cause spike traffic, but even that should be manageable I feel given I have a CDN and w3cache set up. The site runs smoothly with no problem 99% of the time. I've even run a load test using loadimpact.com and the server handled it fine.
Here are some logs from latest crash, maybe someone can get some clues:
Error Log
[Fri Jun 19 03:06:07 2015] [error] server reached MaxClients setting, consider raising the MaxClients setting
MYSQL LOG
150619 03:06:09 mysqld_safe Number of processes running now: 0
150619 03:06:12 mysqld_safe mysqld restarted
150619 3:06:15 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150619 3:06:15 [Note] Plugin 'FEDERATED' is disabled.
150619 3:06:15 InnoDB: The InnoDB memory heap is disabled
150619 3:06:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150619 3:06:15 InnoDB: Compressed tables use zlib 1.2.3
150619 3:06:15 InnoDB: Using Linux native AIO
150619 3:06:16 InnoDB: Initializing buffer pool, size = 320.0M
150619 3:06:16 InnoDB: Completed initialization of buffer pool
150619 3:06:17 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150619 3:06:17 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
SYSLOG MESSAGES
Jun 19 03:06:07 panel kernel: php invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 19 03:06:07 panel kernel: php cpuset=/ mems_allowed=0
Jun 19 03:06:07 panel kernel: CPU: 1 PID: 24844 Comm: php Not tainted 4.0.5-x86_64-linode58 #1
Jun 19 03:06:07 panel kernel: 0000000000000000 ffff880055fac740 ffffffff8193b626 ffff880055fac100
Jun 19 03:06:07 panel kernel: ffffffff819360a4 0100000000000400 ffff88007fc17000 0000000000000020
Jun 19 03:06:07 panel kernel: ffff88007fc17000 0000000000000000 0000000000000000 0000000000000000
Jun 19 03:06:07 panel kernel: Call Trace:
Jun 19 03:06:07 panel kernel: [<ffffffff8193b626>] ? dump_stack+0x40/0x50
Jun 19 03:06:07 panel kernel: [<ffffffff819360a4>] ? dump_header.isra.10+0x78/0x1e3
Jun 19 03:06:07 panel kernel: [<ffffffff81941bb6>] ? _raw_spin_unlock_irqrestore+0x2e/0x3f
Jun 19 03:06:07 panel kernel: [<ffffffff81172f4a>] ? oom_kill_process+0xbe/0x380
Jun 19 03:06:07 panel kernel: [<ffffffff810dc57d>] ? has_ns_capability_noaudit+0x13/0x1b
Jun 19 03:06:07 panel kernel: [<ffffffff8117369a>] ? __out_of_memory+0x43d/0x47d
Jun 19 03:06:07 panel kernel: [<ffffffff8117381b>] ? out_of_memory+0x52/0x67
Jun 19 03:06:07 panel kernel: [<ffffffff811777e5>] ? __alloc_pages_nodemask+0x708/0x846
Jun 19 03:06:07 panel kernel: [<ffffffff811a4a7d>] ? alloc_pages_current+0xb2/0xcf
Jun 19 03:06:07 panel kernel: [<ffffffff8117205d>] ? filemap_fault+0x26f/0x394
Jun 19 03:06:07 panel kernel: [<ffffffff8118f78d>] ? __do_fault+0x3f/0x79
Jun 19 03:06:07 panel kernel: [<ffffffff81193009>] ? handle_mm_fault+0x370/0xd80
Jun 19 03:06:07 panel kernel: [<ffffffff811426a1>] ? __audit_syscall_exit+0x208/0x224
Jun 19 03:06:07 panel kernel: [<ffffffff8104219d>] ? __do_page_fault+0x321/0x37b
Jun 19 03:06:07 panel kernel: [<ffffffff81943f08>] ? page_fault+0x28/0x30
Jun 19 03:06:07 panel kernel: Mem-Info:
Jun 19 03:06:07 panel kernel: Node 0 DMA per-cpu:
Jun 19 03:06:07 panel kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jun 19 03:06:07 panel kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jun 19 03:06:07 panel kernel: Node 0 DMA32 per-cpu:
Jun 19 03:06:07 panel kernel: CPU 0: hi: 186, btch: 31 usd: 3
Jun 19 03:06:07 panel kernel: CPU 1: hi: 186, btch: 31 usd: 0
Jun 19 03:06:07 panel kernel: active_anon:348396 inactive_anon:116687 isolated_anon:32
Jun 19 03:06:07 panel kernel: active_file:351 inactive_file:528 isolated_file:0
Jun 19 03:06:07 panel kernel: unevictable:0 dirty:0 writeback:6 unstable:0
Jun 19 03:06:07 panel kernel: free:3411 slab_reclaimable:8339 slab_unreclaimable:5537
Jun 19 03:06:07 panel kernel: mapped:169339 shmem:168953 pagetables:8887 bounce:0
Jun 19 03:06:07 panel kernel: free_cma:0
Jun 19 03:06:07 panel kernel: Node 0 DMA free:7928kB min:44kB low:52kB high:64kB active_anon:2880kB inactive_anon:3140kB active_file:0kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:2676kB shmem:2664kB slab_reclaimable:44kB slab_unreclaimable:164kB kernel_stack:0kB pagetables:132kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:20 all_unreclaimable? no
Jun 19 03:06:07 panel kernel: lowmem_reserve[]: 0 1972 1972 1972
Jun 19 03:06:07 panel kernel: Node 0 DMA32 free:5736kB min:5656kB low:7068kB high:8484kB active_anon:1390704kB inactive_anon:463608kB active_file:1488kB inactive_file:1892kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:2080768kB managed:2023584kB mlocked:0kB dirty:0kB writeback:24kB mapped:675052kB shmem:673148kB slab_reclaimable:33312kB slab_unreclaimable:21984kB kernel_stack:4304kB pagetables:35416kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 19 03:06:07 panel kernel: lowmem_reserve[]: 0 0 0 0
Jun 19 03:06:07 panel kernel: Node 0 DMA: 4*4kB (UM) 5*8kB (U) 8*16kB (UM) 4*32kB (UM) 3*64kB (UE) 10*128kB (EM) 6*256kB (UEM) 1*512kB (U) 0*1024kB 0*2048kB 1*4096kB (R) = 7928kB
Jun 19 03:06:07 panel kernel: Node 0 DMA32: 97*4kB (UEMR) 10*8kB (MR) 0*16kB 0*32kB 1*64kB (R) 1*128kB (R) 0*256kB 0*512kB 1*1024kB (R) 0*2048kB 1*4096kB (R) = 5780kB
Jun 19 03:06:07 panel kernel: 174507 total pagecache pages
Jun 19 03:06:07 panel kernel: 4598 pages in swap cache
Jun 19 03:06:07 panel kernel: Swap cache stats: add 1673544, delete 1668946, find 6783627/7234079
Jun 19 03:06:07 panel kernel: Free swap = 0kB
Jun 19 03:06:07 panel kernel: Total swap = 262140kB
Jun 19 03:06:07 panel kernel: 524191 pages RAM
Jun 19 03:06:07 panel kernel: 0 pages HighMem/MovableOnly
Jun 19 03:06:07 panel kernel: 14317 pages reserved
Jun 19 03:06:07 panel kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Jun 19 03:06:07 panel kernel: [ 1541] 0 1541 2710 0 9 2 127 -1000 udevd
Jun 19 03:06:07 panel kernel: [ 2129] 0 2129 2295 34 8 2 90 0 dhclient
Jun 19 03:06:07 panel kernel: [ 2175] 0 2175 6402 0 12 2 75 -1000 auditd
Jun 19 03:06:07 panel kernel: [ 2238] 0 2238 60749 0 21 3 214 0 rsyslogd
Jun 19 03:06:07 panel kernel: [ 2264] 25 2264 57836 0 44 2 4143 0 named
Jun 19 03:06:07 panel kernel: [ 2295] 81 2295 5373 0 14 2 63 0 dbus-daemon
Jun 19 03:06:07 panel kernel: [ 2350] 495 2350 80168 0 26 5 86 0 memcached
Jun 19 03:06:07 panel kernel: [ 2368] 0 2368 16569 11 34 2 166 -1000 sshd
Jun 19 03:06:07 panel kernel: [ 2662] 0 2662 4943 23 19 5 54 0 dovecot
Jun 19 03:06:07 panel kernel: [ 2665] 498 2665 10600 0 41 6 144 0 pop3-login
Jun 19 03:06:07 panel kernel: [ 2666] 498 2666 10639 0 41 6 153 0 imap-login
Jun 19 03:06:07 panel kernel: [ 2667] 97 2667 3369 0 18 5 49 0 anvil
Jun 19 03:06:07 panel kernel: [ 2668] 0 2668 3402 0 18 5 78 0 log
Jun 19 03:06:07 panel kernel: [ 2671] 498 2671 10636 18 41 6 120 0 pop3-login
Jun 19 03:06:07 panel kernel: [ 2672] 0 2672 4145 0 19 5 282 0 config
Jun 19 03:06:07 panel kernel: [ 2673] 498 2673 10635 0 42 6 145 0 imap-login
Jun 19 03:06:07 panel kernel: [ 2683] 47 2683 18053 15 49 7 136 0 exim
Jun 19 03:06:07 panel kernel: [ 2807] 0 2807 43014 217 108 6 12952 0 /usr/local/cpan
Jun 19 03:06:07 panel kernel: [ 2895] 0 2895 43249 4306 97 6 9070 0 spamd child
Jun 19 03:06:07 panel kernel: [ 2952] 501 2952 4042 0 14 4 33 0 newrelic-daemon
Jun 19 03:06:07 panel kernel: [ 2953] 0 2953 28675 0 16 2 52 0 abrtd
Jun 19 03:06:07 panel kernel: [ 2954] 501 2954 60989 123 29 4 544 0 newrelic-daemon
Jun 19 03:06:07 panel kernel: [ 2970] 0 2970 28181 0 23 4 65 0 abrt-dump-oops
Jun 19 03:06:07 panel kernel: [ 2981] 0 2981 34674 9 23 2 160 0 pure-ftpd
Jun 19 03:06:07 panel kernel: [ 2983] 0 2983 34047 7 27 2 114 0 pure-authd
Jun 19 03:06:07 panel kernel: [ 2995] 0 2995 29217 8 16 2 148 0 crond
Jun 19 03:06:07 panel kernel: [ 3009] 0 3009 5291 11 14 2 34 0 atd
Jun 19 03:06:07 panel kernel: [ 3521] 0 3521 35818 389 90 6 6493 0 cpsrvd (SSL) -
Jun 19 03:06:07 panel kernel: [ 3760] 0 3760 10172 158 34 4 1275 0 queueprocd - wa
Jun 19 03:06:07 panel kernel: [ 3832] 0 3832 28548 0 66 5 4769 0 dnsadmin - serv
Jun 19 03:06:07 panel kernel: [ 3858] 0 3858 21764 452 54 5 3245 0 tailwatchd
Jun 19 03:06:07 panel kernel: [ 3934] 0 3934 29015 2 65 5 5416 0 cpdavd - accept
Jun 19 03:06:07 panel kernel: [ 3964] 0 3964 7951 11 27 4 1315 0 cpanellogd - sl
Jun 19 03:06:07 panel kernel: [ 4002] 0 4002 1035 1 9 4 19 0 agetty
Jun 19 03:06:07 panel kernel: [ 4004] 0 4004 1031 0 9 4 17 0 mingetty
Jun 19 03:06:07 panel kernel: [ 4006] 0 4006 1031 0 9 4 18 0 mingetty
Jun 19 03:06:07 panel kernel: [ 4008] 0 4008 1031 0 9 4 18 0 mingetty
Jun 19 03:06:07 panel kernel: [ 4011] 0 4011 2709 0 9 2 128 -1000 udevd
Jun 19 03:06:07 panel kernel: [ 4012] 0 4012 2709 0 9 2 128 -1000 udevd
Jun 19 03:06:07 panel kernel: [ 4013] 0 4013 1031 0 9 4 18 0 mingetty
Jun 19 03:06:07 panel kernel: [ 4015] 0 4015 1031 0 9 4 18 0 mingetty
Jun 19 03:06:07 panel kernel: [ 4017] 0 4017 1031 0 8 4 17 0 mingetty
Jun 19 03:06:07 panel kernel: [16053] 0 16053 18080 444 59 5 138 0 httpd
Jun 19 03:06:07 panel kernel: [19702] 0 19702 2883 1 15 4 71 0 mysqld_safe
Jun 19 03:06:07 panel kernel: [19964] 497 19964 393396 49834 210 7 17743 0 mysqld
Jun 19 03:06:07 panel kernel: [22735] 0 22735 3368 26 17 5 0 0 ssl-params
Jun 19 03:06:07 panel kernel: [24373] 0 24373 20166 2064 63 5 9 0 leechprotect
Jun 19 03:06:07 panel kernel: [24374] 99 24374 18247 625 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24375] 99 24375 18259 641 48 5 127 0 httpd
Jun 19 03:06:07 panel kernel: [24378] 99 24378 18283 650 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24379] 99 24379 18207 582 48 5 131 0 httpd
Jun 19 03:06:07 panel kernel: [24380] 99 24380 18294 651 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24383] 99 24383 18251 632 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24388] 99 24388 18275 651 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24395] 99 24395 18232 610 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24438] 0 24438 4252 51 12 2 0 0 anacron
Jun 19 03:06:07 panel kernel: [24457] 99 24457 18249 629 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24460] 99 24460 18243 622 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24559] 99 24559 18234 611 48 5 129 0 httpd
Jun 19 03:06:07 panel kernel: [24688] 99 24688 18283 646 48 5 131 0 httpd
Jun 19 03:06:07 panel kernel: [24757] 97 24757 7930 113 30 5 0 0 auth
Jun 19 03:06:07 panel kernel: [24838] 507 24838 84744 21633 134 6 0 0 php
Jun 19 03:06:07 panel kernel: [24840] 507 24840 83776 20878 133 6 0 0 php
Jun 19 03:06:07 panel kernel: [24841] 507 24841 82483 18820 128 7 0 0 php
Jun 19 03:06:07 panel kernel: [24844] 507 24844 82937 20101 130 7 0 0 php
Jun 19 03:06:07 panel kernel: [24845] 507 24845 82227 18788 129 7 0 0 php
Jun 19 03:06:07 panel kernel: [24846] 507 24846 82291 19040 129 6 0 0 php
Jun 19 03:06:07 panel kernel: [24847] 507 24847 82887 19792 131 7 0 0 php
Jun 19 03:06:07 panel kernel: [24848] 507 24848 82934 20094 130 7 0 0 php
Jun 19 03:06:07 panel kernel: [24850] 507 24850 82894 19887 130 6 0 0 php
Jun 19 03:06:07 panel kernel: [24852] 507 24852 82548 19709 129 6 0 0 php
Jun 19 03:06:07 panel kernel: [24859] 507 24859 82803 19724 131 6 0 0 php
Jun 19 03:06:07 panel kernel: [24862] 99 24862 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24866] 507 24866 81465 17804 126 6 0 0 php
Jun 19 03:06:07 panel kernel: [24875] 99 24875 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24876] 99 24876 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24877] 507 24877 79415 14127 121 6 0 0 php
Jun 19 03:06:07 panel kernel: [24878] 507 24878 79221 13720 118 7 0 0 php
Jun 19 03:06:07 panel kernel: [24885] 99 24885 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24886] 99 24886 18182 523 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24887] 99 24887 18182 523 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24888] 99 24888 18182 523 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24892] 507 24892 77328 9824 111 6 0 0 php
Jun 19 03:06:07 panel kernel: [24898] 99 24898 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24899] 99 24899 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24900] 99 24900 18182 513 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24901] 99 24901 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24902] 99 24902 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24903] 99 24903 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24904] 99 24904 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24905] 99 24905 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24906] 507 24906 75260 6007 103 6 0 0 php
Jun 19 03:06:07 panel kernel: [24907] 507 24907 61544 14697 114 6 0 0 php
Jun 19 03:06:07 panel kernel: [24908] 507 24908 75724 6894 105 6 0 0 php
Jun 19 03:06:07 panel kernel: [24909] 507 24909 74348 5097 101 6 0 0 php
Jun 19 03:06:07 panel kernel: [24910] 507 24910 75746 6708 104 7 0 0 php
Jun 19 03:06:07 panel kernel: [24911] 507 24911 75746 6630 104 6 0 0 php
Jun 19 03:06:07 panel kernel: [24912] 507 24912 75746 6705 104 6 0 0 php
Jun 19 03:06:07 panel kernel: [24913] 507 24913 76743 8656 108 6 0 0 php
Jun 19 03:06:07 panel kernel: [24922] 507 24922 73138 3847 99 7 0 0 php
Jun 19 03:06:07 panel kernel: [24923] 507 24923 72900 3513 96 6 0 0 php
Jun 19 03:06:07 panel kernel: [24925] 507 24925 72704 3279 98 6 0 0 php
Jun 19 03:06:07 panel kernel: [24926] 507 24926 72856 3395 98 6 0 0 php
Jun 19 03:06:07 panel kernel: [24927] 99 24927 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24928] 99 24928 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24929] 99 24929 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24930] 99 24930 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24931] 99 24931 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24932] 99 24932 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24933] 99 24933 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24934] 99 24934 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24935] 99 24935 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24936] 507 24936 72900 3454 98 7 0 0 php
Jun 19 03:06:07 panel kernel: [24937] 507 24937 72900 3512 100 6 0 0 php
Jun 19 03:06:07 panel kernel: [24938] 507 24938 72898 3510 99 6 0 0 php
Jun 19 03:06:07 panel kernel: [24940] 99 24940 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24941] 99 24941 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24942] 99 24942 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24944] 507 24944 72709 3228 98 6 0 0 php
Jun 19 03:06:07 panel kernel: [24945] 507 24945 72707 3210 96 6 0 0 php
Jun 19 03:06:07 panel kernel: [24946] 507 24946 72505 2756 97 6 0 0 php
Jun 19 03:06:07 panel kernel: [24947] 507 24947 72770 3394 99 6 0 0 php
Jun 19 03:06:07 panel kernel: [24948] 507 24948 72898 3511 98 7 0 0 php
Jun 19 03:06:07 panel kernel: [24950] 507 24950 72920 3491 99 6 0 0 php
Jun 19 03:06:07 panel kernel: [24952] 99 24952 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24953] 99 24953 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24954] 99 24954 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24956] 507 24956 72273 2332 95 6 0 0 php
Jun 19 03:06:07 panel kernel: [24959] 507 24959 71635 1659 96 6 0 0 php
Jun 19 03:06:07 panel kernel: [24960] 507 24960 72358 2601 97 7 0 0 php
Jun 19 03:06:07 panel kernel: [24961] 507 24961 72431 2797 96 6 0 0 php
Jun 19 03:06:07 panel kernel: [24962] 507 24962 72565 2478 97 6 0 0 php
Jun 19 03:06:07 panel kernel: [24963] 99 24963 18149 480 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24964] 507 24964 72505 2727 98 6 0 0 php
Jun 19 03:06:07 panel kernel: [24965] 507 24965 71871 1883 95 6 0 0 php
Jun 19 03:06:07 panel kernel: [24987] 99 24987 18149 467 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24988] 99 24988 18149 472 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24989] 99 24989 18149 467 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24990] 99 24990 18149 479 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24991] 99 24991 18149 477 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24992] 507 24992 24451 62 72 6 0 0 php
Jun 19 03:06:07 panel kernel: [24993] 99 24993 18149 478 48 5 132 0 httpd
Jun 19 03:06:07 panel kernel: [24994] 99 24994 2998 13 14 4 0 0 suphp
Jun 19 03:06:07 panel kernel: [24995] 507 24995 5810 9 12 4 0 0 php
Jun 19 03:06:07 panel kernel: [24996] 507 24996 3880 9 10 4 0 0 php
Jun 19 03:06:07 panel kernel: [24998] 507 24998 6373 16 15 5 0 0 php
Jun 19 03:06:07 panel kernel: [24999] 507 24999 2767 1 3 3 0 0 php
Jun 19 03:06:07 panel kernel: [25000] 0 25000 18080 432 47 5 135 0 httpd
Jun 19 03:06:07 panel kernel: Out of memory: Kill process 19964 (mysqld) score 117 or sacrifice child
Jun 19 03:06:07 panel kernel: Killed process 19964 (mysqld) total-vm:1573584kB, anon-rss:199336kB, file-rss:0kB
Jun 19 03:06:07 panel kernel: php invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
Jun 19 03:06:07 panel kernel: php cpuset=/ mems_allowed=0
ANd here's my apache settings:
Starts Servers 2
Minimum Spare Servers 6
Maximum Spare Servers 12
Server Limit 256
Max Clients 50
Max Requests Per Child 3000
Keep-Alive Off
Mysql my.cnf settings:
[mysqld]
innodb_file_per_table=1
open_files_limit=1024000
max_connections=75
key_buffer=32M
max_allowed_packet=268435456
thread_stack=128K
table_cache=32
innodb_buffer_pool_size=320M
default-storage-engine=MyISAM
query_cache_size=8M
tmp_table_size=16M
max_heap_table_size=16M
thread_cache_size=4
table_open_cache=32
Please help, I've tried my best to track this thing down but I can't really pinpoint what is causing it.
BTW some additional random tidbits:
-Some of the pages on my site have LOTS of pictures (like over 100), not sure if these count as extra clients/requests and affecting performance somehow.
-My WordPress memory limit is set around 400MB.
-Mysql database for this site is around 175MB. 91MB of it coming from WP_Options.
-Mysqltuner.pl keeps telling me to increase tmp_table_size and max_heap_table_size even though I've set them as high as 200M!
-Here's a screenshot of my htop right now. Seems mysql is constantly using 40% of memory and Swap memory is almost full for some reason. Is this normal? :
UPDATE: I've lowered Max Clients to 35 but still getting some weird mysql crash. Newrelic error message is Unknown storage engine 'InnoDB' (my mysqltuner.pl is saying InnoDB is disabled but that it's default storage engine). My site is telling me to reinstall WordPress like a fresh install, and I have to restart mysql or the server for that to go away.
Best Answer
...but your screenshot shows 23 active mysql sessions, and you configured your webserver to allow 256.
Swap on cheap hosting is an anti-pattern - once your machine starts getting into swap it will spend all its time reading and writing the swap file while the requests start backing up on Apache.
I have nothing good to say about Wordpress.
I see from your screenshot that the CLI SAPI for PHP is running. And oom killer decided that PHP was the worst offender for memory on the system. That suggests you are running CGI PHP rather than mod_php or php-fpm. If that is the case, then we've not only solved the problem of why your host is running out of memory, but also why your pages are sooooooooo sloooooooooooowwwww,
That's really not helping. It's certainly making the experience at the browser end much worse. It should be on with a timeout below 500ms even if you have got your caching and CDN configured exactly right.
That's obviously not the entirety of your Apache config - and a lot of the other settings are important too. Ensuring compression is ENABLED is a big one (it's less expensive in CPU time to compress the data than to handle the TCP traffic and means you don't have big apache processes hanging around in memory waiting for data transfers).
Although you provided lots of information about what happened after the oom killer you need to look at what is happening before that. And that information is not captured by default. A minimum would be to track the memory usage vs number of apache instances.