Can you please tell me what cause this Oracle process killed? Seems like plenty of RAM free, and plenty of SWAP free. There followed few other oracle processes killed.
The VM has the 16G of vMem and 8 vCPU.
But I am posting here the first oracle process that got killed:
Mar 1 20:00:58 ******* kernel: oracle invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 Mar 1 20:00:58 ******* kernel: oracle cpuset=/ mems_allowed=0 Mar 1 20:00:58 ******* kernel: Pid: 2370, comm: oracle Not tainted 2.6.32-431.el6.x86_64 #1 Mar 1 20:00:58 ******* kernel: Call Trace: Mar 1 20:00:58 ******* kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0 Mar 1 20:00:58 ******* kernel: [] ? dump_header+0x90/0x1b0 Mar 1 20:00:58 ******* kernel: [] ? security_real_capable_noaudit+0x3c/0x70 Mar 1 20:00:58 ******* kernel: [] ? oom_kill_process+0x82/0x2a0 Mar 1 20:00:58 ******* kernel: [] ? select_bad_process+0xe1/0x120 Mar 1 20:00:58 ******* kernel: [] ? out_of_memory+0x220/0x3c0 Mar 1 20:00:58 ******* kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0 Mar 1 20:00:58 ******* kernel: [] ? alloc_pages_vma+0x9a/0x150 Mar 1 20:00:58 ******* kernel: [] ? handle_pte_fault+0x73d/0xb00 Mar 1 20:00:58 ******* kernel: [] ? free_pgtables+0xce/0x120 Mar 1 20:00:58 ******* kernel: [] ? unmap_region+0xcd/0x130 Mar 1 20:00:58 ******* kernel: [] ? vma_prio_tree_add+0x75/0xd0 Mar 1 20:00:58 ******* kernel: [] ? handle_mm_fault+0x22a/0x300 Mar 1 20:00:58 ******* kernel: [] ? __do_page_fault+0x138/0x480 Mar 1 20:00:58 ******* kernel: [] ? do_mmap_pgoff+0x335/0x380 Mar 1 20:00:58 ******* kernel: [] ? do_page_fault+0x3e/0xa0 Mar 1 20:00:58 ******* kernel: [] ? page_fault+0x25/0x30 Mar 1 20:00:58 ******* kernel: Mem-Info: Mar 1 20:00:58 ******* kernel: Node 0 DMA per-cpu: Mar 1 20:00:58 ******* kernel: CPU 0: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 1: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 2: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 3: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 4: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 5: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 6: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 7: hi: 0, btch: 1 usd: 0 Mar 1 20:00:58 ******* kernel: Node 0 DMA32 per-cpu: Mar 1 20:00:58 ******* kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 2: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 3: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 4: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 5: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 6: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 7: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: Node 0 Normal per-cpu: Mar 1 20:00:58 ******* kernel: CPU 0: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 1: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 2: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 3: hi: 186, btch: 31 usd: 20 Mar 1 20:00:58 ******* kernel: CPU 4: hi: 186, btch: 31 usd: 32 Mar 1 20:00:58 ******* kernel: CPU 5: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: CPU 6: hi: 186, btch: 31 usd: 184 Mar 1 20:00:58 ******* kernel: CPU 7: hi: 186, btch: 31 usd: 0 Mar 1 20:00:58 ******* kernel: active_anon:2673615 inactive_anon:368657 isolated_anon:0 Mar 1 20:00:58 ******* kernel: active_file:3541 inactive_file:3962 isolated_file:32 Mar 1 20:00:58 ******* kernel: unevictable:0 dirty:3 writeback:2770 unstable:0 Mar 1 20:00:58 ******* kernel: free:33763 slab_reclaimable:16555 slab_unreclaimable:28221 Mar 1 20:00:58 ******* kernel: mapped:1517627 shmem:1730877 pagetables:906135 bounce:0 Mar 1 20:00:58 ******* kernel: Node 0 DMA free:15132kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14740kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Mar 1 20:00:58 ******* kernel: lowmem_reserve[]: 0 3000 16130 16130 Mar 1 20:00:58 ******* kernel: Node 0 DMA32 free:64904kB min:12556kB low:15692kB high:18832kB active_anon:2064816kB inactive_anon:516452kB active_file:492kB inactive_file:188kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:0kB mapped:2319432kB shmem:2352892kB slab_reclaimable:7420kB slab_unreclaimable:3620kB kernel_stack:832kB pagetables:24672kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1 all_unreclaimable? no Mar 1 20:00:58 ******* kernel: lowmem_reserve[]: 0 0 13130 13130 Mar 1 20:00:58 ******* kernel: Node 0 Normal free:55016kB min:54964kB low:68704kB high:82444kB active_anon:8629644kB inactive_anon:958176kB active_file:13672kB inactive_file:15660kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:13445120kB mlocked:0kB dirty:12kB writeback:11080kB mapped:3751076kB shmem:4570616kB slab_reclaimable:58800kB slab_unreclaimable:109264kB kernel_stack:5360kB pagetables:3599868kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? no Mar 1 20:00:58 ******* kernel: lowmem_reserve[]: 0 0 0 0 Mar 1 20:00:58 ******* kernel: Node 0 DMA: 3*4kB 2*8kB 2*16kB 3*32kB 2*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 3*4096kB = 15132kB Mar 1 20:00:58 ******* kernel: Node 0 DMA32: 1225*4kB 859*8kB 878*16kB 547*32kB 184*64kB 34*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 65596kB Mar 1 20:00:58 ******* kernel: Node 0 Normal: 9165*4kB 1804*8kB 46*16kB 2*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 55924kB Mar 1 20:00:58 ******* kernel: 1760824 total pagecache pages Mar 1 20:00:58 ******* kernel: 22460 pages in swap cache Mar 1 20:00:58 ******* kernel: Swap cache stats: add 6636857, delete 6614397, find 15635455/16141480 Mar 1 20:00:58 ******* kernel: Free swap = 33548340kB Mar 1 20:00:58 ******* kernel: Total swap = 36184056kB Mar 1 20:00:58 ******* kernel: 4194288 pages RAM Mar 1 20:00:58 ******* kernel: 111808 pages reserved Mar 1 20:00:58 ******* kernel: 59252583 pages shared Mar 1 20:00:58 ******* kernel: 2502605 pages non-shared Mar 1 20:00:58 ******* kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Mar 1 20:00:58 ******* kernel: [ 612] 0 612 2769 42 2 -17 -1000 udevd Mar 1 20:00:58 ******* kernel: [ 1872] 0 1872 47365 204 7 0 0 vmtoolsd Mar 1 20:00:58 ******* kernel: [ 1980] 0 1980 23294 109 6 -17 -1000 auditd Mar 1 20:00:58 ******* kernel: [ 1996] 0 1996 62898 842 4 0 0 rsyslogd Mar 1 20:00:58 ******* kernel: [ 2025] 0 2025 2738 93 3 0 0 irqbalance Mar 1 20:00:58 ******* kernel: [ 2039] 32 2039 4744 68 4 0 0 rpcbind Mar 1 20:00:58 ******* kernel: [ 2071] 29 2071 5837 61 3 0 0 rpc.statd Mar 1 20:00:58 ******* kernel: [ 2092] 0 2092 5773 31 1 0 0 rpc.idmapd Mar 1 20:00:58 ******* kernel: [ 2211] 0 2211 39323 127 5 0 0 pbx_exchange Mar 1 20:00:58 ******* kernel: [ 2223] 0 2223 48106 158 5 0 0 winbindd Mar 1 20:00:58 ******* kernel: [ 2237] 0 2237 1020 48 4 0 0 acpid Mar 1 20:00:58 ******* kernel: [ 2323] 0 2323 49766 281 0 0 0 winbindd Mar 1 20:00:58 ******* kernel: [ 2540] 0 2540 26827 11 5 0 0 rpc.rquotad Mar 1 20:00:58 ******* kernel: [ 2544] 0 2544 5414 41 5 0 0 rpc.mountd Mar 1 20:00:58 ******* kernel: [ 2580] 0 2580 1570 23 0 0 0 mcelog Mar 1 20:00:58 ******* kernel: [ 2592] 0 2592 16651 78 5 -17 -1000 sshd Mar 1 20:00:58 ******* kernel: [ 2600] 0 2600 5545 105 3 0 0 xinetd Mar 1 20:00:58 ******* kernel: [ 2608] 38 2608 7147 132 5 0 0 ntpd Mar 1 20:00:58 ******* kernel: [ 2618] 498 2618 25741 57 2 0 0 uuidd Mar 1 20:00:58 ******* kernel: [ 2630] 0 2630 43170 139 3 0 0 vnetd Mar 1 20:00:58 ******* kernel: [ 2638] 0 2638 52398 158 2 0 0 bpcd Mar 1 20:00:58 ******* kernel: [ 2655] 0 2655 198335 478 4 0 0 nbdisco Mar 1 20:00:58 ******* kernel: [ 2676] 0 2676 76958 82 2 0 0 mtstrmd Mar 1 20:00:58 ******* kernel: [ 2707] 0 2707 22314 141 0 0 0 sendmail Mar 1 20:00:58 ******* kernel: [ 2716] 51 2716 19658 80 4 0 0 sendmail Mar 1 20:00:58 ******* kernel: [ 2734] 0 2734 200856 353 7 0 0 avagent.bin Mar 1 20:00:58 ******* kernel: [ 2747] 0 2747 44287 178 3 0 0 tuned Mar 1 20:00:58 ******* kernel: [ 2757] 0 2757 29333 103 6 0 0 crond Mar 1 20:00:58 ******* kernel: [ 2778] 0 2778 27431 167 7 0 0 saphostexec Mar 1 20:00:58 ******* kernel: [ 2805] 600 2805 545016 4031 5 0 0 sapstartsrv Mar 1 20:00:58 ******* kernel: [ 2885] 834 2885 100602 294 3 0 0 sapstartsrv Mar 1 20:00:58 ******* kernel: [ 2904] 0 2904 5385 31 6 0 0 atd Mar 1 20:00:58 ******* kernel: [ 2928] 0 2928 26005 69 5 0 0 rhsmcertd Mar 1 20:00:58 ******* kernel: [ 2935] 0 2935 8154 1110 0 0 0 saposcol Mar 1 20:00:58 ******* kernel: [ 3098] 834 3098 13538 50 3 0 0 sapstart Mar 1 20:00:58 ******* kernel: [ 3128] 834 3128 43278 119 5 0 0 jc.sapDAA_SMDA9 Mar 1 20:00:58 ******* kernel: [ 3144] 834 3144 1276839 57796 4 0 0 jstart Mar 1 20:00:58 ******* kernel: [ 3211] 703 3211 33752 378 5 0 0 perl Mar 1 20:00:58 ******* kernel: [ 3288] 703 3288 1181563 62355 0 0 0 java Mar 1 20:00:58 ******* kernel: [ 3497] 0 3497 1016 34 1 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3499] 0 3499 1016 34 1 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3502] 0 3502 1016 34 1 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3504] 0 3504 1016 34 2 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3506] 0 3506 1016 34 1 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3508] 0 3508 1016 34 1 0 0 mingetty Mar 1 20:00:58 ******* kernel: [ 3515] 0 3515 3098 41 2 -17 -1000 udevd Mar 1 20:00:58 ******* kernel: [ 3516] 0 3516 3098 41 4 -17 -1000 udevd Mar 1 20:00:58 ******* kernel: [13764] 0 13764 48089 89 7 0 0 winbindd Mar 1 20:00:58 ******* kernel: [13765] 0 13765 48089 92 7 0 0 winbindd Mar 1 20:00:58 ******* kernel: [13873] 703 13873 2403434 6196 5 0 0 oracle Mar 1 20:00:58 ******* kernel: [13875] 703 13875 2402873 651 3 0 0 oracle Mar 1 20:00:58 ******* kernel: [13880] 703 13880 2402873 423 4 0 0 oracle Mar 1 20:00:58 ******* kernel: [13875] 703 13875 2402873 651 3 0 0 oracle Mar 1 20:00:58 ******* kernel: [13880] 703 13880 2402873 423 4 0 0 oracle .. Note: Removed bunch of oracle processes here so as to limit the character length for the posting here. Total of 296 oracle process running. .. Mar 1 20:00:59 ******* kernel: [18644] 0 18644 44207 371 1 0 0 bpclntcmd Mar 1 20:00:59 ******* kernel: [18647] 703 18647 57442 240 3 0 0 oracle Mar 1 20:00:59 ******* kernel: [18656] 703 18656 57442 185 6 0 0 oracle Mar 1 20:00:59 ******* kernel: [18657] 54329 18657 9279 196 1 0 0 nrpe Mar 1 20:00:59 ******* kernel: [18660] 54329 18660 9314 255 2 0 0 nrpe Mar 1 20:00:59 ******* kernel: [18662] 0 18662 39263 289 5 0 0 crond Mar 1 20:00:59 ******* kernel: [18663] 0 18663 5745 341 1 0 0 saposcol Mar 1 20:00:59 ******* kernel: [18664] 54329 18664 9315 146 3 0 0 nrpe Mar 1 20:00:59 ******* kernel: [18665] 54329 18665 5730 76 0 0 0 check_open_file Mar 1 20:00:59 ******* kernel: [18666] 54329 18666 6611 191 4 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18667] 0 18667 8389 183 1 0 0 sapcimb Mar 1 20:00:59 ******* kernel: [18669] 0 18669 6610 171 0 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18670] 0 18670 6610 171 0 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18677] 0 18677 6610 177 5 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18678] 703 18678 29497 275 4 0 0 perl Mar 1 20:00:59 ******* kernel: [18682] 703 18682 29497 252 7 0 0 perl Mar 1 20:00:59 ******* kernel: [18683] 703 18683 29497 231 0 0 0 perl Mar 1 20:00:59 ******* kernel: [18687] 0 18687 2620 92 1 0 0 .SAPOSCOL_00000 Mar 1 20:00:59 ******* kernel: [18688] 0 18688 6610 186 5 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18689] 0 18689 6610 189 2 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18690] 0 18690 6610 191 3 0 0 xinetd Mar 1 20:00:59 ******* kernel: [18691] 0 18691 6610 194 2 0 0 xinetd Mar 1 20:00:59 ******* kernel: Out of memory: Kill process 13900 (oracle) score 77 or sacrifice child Mar 1 20:00:59 ******* kernel: Killed process 13900, UID 703, (oracle) total-vm:9622308kB, anon-rss:5180kB, file-rss:4028040kB
From above, I think these lines says I have plenty of RAM and swap. right?:
Node 0 DMA free:15132kB Node 0 DMA32 free:64904kB Node 0 Normal free:55016kB Free swap = 33548340kB Total swap = 36184056kB
Wondering what does it mean by "all_unreclaimable? yes" for Node 0 DMA, and "all_unreclaimable? no"Node 0 DMA31 and Node 0 Normal !
Also, here are the info that might give more info about the server settings:
$sudo sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key error: "net.bridge.bridge-nf-call-iptables" is an unknown key error: "net.bridge.bridge-nf-call-arptables" is an unknown key kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.msgmni = 1024 kernel.sem = 1250 256000 100 8192 vm.max_map_count = 1000000 kernel.shmall = 1152921504606846720 fs.file-max = 19801952 net.core.rmem_default = 1048576 net.core.wmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_max = 1048576 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 vm.swappiness = 0 vm.dirty_background_ratio = 3 vm.dirty_ratio = 15 vm.dirty_expire_centisecs = 500 vm.dirty_writeback_centisecs = 100 kernel.shmmni = 4096
Best Answer
You don't have much free memory at all.
First,
vm.swappiness = 0Only do this if you are definitely sure you have enough. Setting it low to 10 or so might prevent an out of memory condition. And will actually make use of your paging space.From the summary of node 0, your 16 GB is roughly a quarter page tables, a quarter shared memory, half anonymous program memory, and some various odds and ends. Notice that the readily available file memory, plus free, is only tens of MB, not large. It won't be able to give you another GB or so of shared memory.
Page tables are eating you alive. You may not have huge pages enabled, which Oracle recommends for databases, and Red Hat does too.