Docker – How does Docker deal with OOM killer and memory limits

dockeroom-killer

I have a docker container which starts a simple java (jgroups-based) application through a bash script. The java process is limited through Xmx by 128m, the container is allowed to use 256m (swap is disabled). Unfortunately, time to time I face the following OOM messages:

Jul 07 02:43:54 ip-10-1-2-125 kernel: oom_kill_process: 16 callbacks suppressed
Jul 07 02:43:54 ip-10-1-2-125 kernel: java invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
Jul 07 02:43:54 ip-10-1-2-125 kernel: java cpuset=0ead341e639c2f2bd27a38666aa0834c969e8c7e6d2fb21516a2c698adce8d5f mems_allowed=0
Jul 07 02:43:54 ip-10-1-2-125 kernel: CPU: 0 PID: 26686 Comm: java Not tainted 4.4.0-28-generic #47-Ubuntu
Jul 07 02:43:54 ip-10-1-2-125 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 05/12/2016
Jul 07 02:43:54 ip-10-1-2-125 kernel:  0000000000000286 000000006ffe9d71 ffff8800bb3c7c88 ffffffff813eb1a3
Jul 07 02:43:54 ip-10-1-2-125 kernel:  ffff8800bb3c7d68 ffff880033aea940 ffff8800bb3c7cf8 ffffffff812094fe
Jul 07 02:43:54 ip-10-1-2-125 kernel:  000000000000258c 000000000000000a ffffffff81e66760 0000000000000206
Jul 07 02:43:54 ip-10-1-2-125 kernel: Call Trace:
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff813eb1a3>] dump_stack+0x63/0x90
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff812094fe>] dump_header+0x5a/0x1c5
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff811913b2>] oom_kill_process+0x202/0x3c0
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff811fd304>] ? mem_cgroup_iter+0x204/0x390
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff811ff363>] mem_cgroup_out_of_memory+0x2b3/0x300
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff81200138>] mem_cgroup_oom_synchronize+0x338/0x350
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff811fb660>] ? kzalloc_node.constprop.48+0x20/0x20
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff81191a64>] pagefault_out_of_memory+0x44/0xc0
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff8106b2c2>] mm_fault_error+0x82/0x160
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff8106b778>] __do_page_fault+0x3d8/0x400
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff8106b7c2>] do_page_fault+0x22/0x30
Jul 07 02:43:54 ip-10-1-2-125 kernel:  [<ffffffff81829838>] page_fault+0x28/0x30
Jul 07 02:43:54 ip-10-1-2-125 kernel: Task in /docker/0ead341e639c2f2bd27a38666aa0834c969e8c7e6d2fb21516a2c698adce8d5f killed as a result of limit of /docker/0ead341e639c2f2bd27a38666aa0834c96
Jul 07 02:43:54 ip-10-1-2-125 kernel: memory: usage 262144kB, limit 262144kB, failcnt 6868
Jul 07 02:43:54 ip-10-1-2-125 kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Jul 07 02:43:54 ip-10-1-2-125 kernel: kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Jul 07 02:43:54 ip-10-1-2-125 kernel: Memory cgroup stats for /docker/0ead341e639c2f2bd27a38666aa0834c969e8c7e6d2fb21516a2c698adce8d5f: cache:96KB rss:262048KB rss_huge:135168KB mapped_file:16
Jul 07 02:43:54 ip-10-1-2-125 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jul 07 02:43:54 ip-10-1-2-125 kernel: [26659]     0 26659     1127       20       7       3        0             0 sh
Jul 07 02:43:54 ip-10-1-2-125 kernel: [26665]     0 26665     1127       20       7       3        0             0 run.sh
Jul 07 02:43:54 ip-10-1-2-125 kernel: [26675]     0 26675   688639    64577     204       7        0             0 java
Jul 07 02:43:54 ip-10-1-2-125 kernel: Memory cgroup out of memory: Kill process 26675 (java) score 988 or sacrifice child
Jul 07 02:43:54 ip-10-1-2-125 kernel: Killed process 26675 (java) total-vm:2754556kB, anon-rss:258308kB, file-rss:0kB
Jul 07 02:43:54 ip-10-1-2-125 docker[977]: Killed

As you can see, RSS of my app is just about 64M. But for some reason RSS of the cgroup is 256M (including 128M of huge pages).

Is that a kind of OS caches? If so, why OOM doesn't flush them before killing user's apps?

Best Answer

Oh! Looks I forgot to post the answer.

The problem above is with my java process, it's not related to docker. I mistakenly thought that OOM report prints RSS in Kbytes. That's wrong - OOM report prints the amount of pages, which are usually take 4K each.

In my case, pid 26675 takes 64577 pages for RSS which equals (64577 * 4K) 258'308 KBytes. Adding 2 bash processes gives us the limit of the current CGroup - 262144kB.

So, the further analysis must be in the JVM field: heap/metaspace analyses, native memory tracking, threads, etc...

Related Solutions

Linux OOM-kill why

Well, I think your min_free_kbytes is really high. I have a 16GB machine and my min is 67584kB.

Note that vmware's ram counts as cache, because of the mmap-ed vmem

Thats not always true. Only if the mmapped() file is opened in MAP_SHARED is that true. Else dirty pages are swap-backed. Which is the case for you it seems. If you add up the reported usage of that process given at the bottom of your output and convert it into pages (4k). It equals the RSS reported in the task dump for that process.

rss:74020kB, file-rss:3099352kB
74020 + 3099352 = 3173372
3173372 / 4 = 793343

is equal to ..

[19635]   501 19635  1693624   793343   1       0        0 vmware-vmx

As for why you OOM-kill. Well, thats a little bit more tricky.

When you reach min the kernel wants to recover memory up to high watermark bytes. The kernel thus has a check; if the amount of memory available to reclaim from the file cache will not be sufficient to put you back into the high watermark of that zone, it wont bother freeing file cache and go straight to reclaiming from anonymous memory.

We never reclaim from active. So -

if (file_inactive > zone_high - free_mem) then
   reclaim (zone_high - free_mem) file inactive pages
else
   reclaim from anonymous pool

In you're case that is 55220 is not greater than 228684-152456 (76428).

The reason this is an OOM-Kill and not swapping is because when you breach the min watermark the kernel goes into a direct_reclaim mode. In this mode, doing IO to free memory cannot be accomplished because it can cause a deadlock.

You're host would have been swapping at the time, but you're host has been allocating faster than it can swap out.

The best way to fix this would be to reduce your min watermark to something lower -- or better still get more memory and/or reduce the amount of things you run on the machine.

Mysql crashing, oom-killer, out of memory, tuning issues

You have hardly any swap (256M), as a temporary measure, I would add more swap and turn swappiness (vm.swappiness) down so as to avoid useless I/O wait. SWAP is slow, but it can keep your PIDS from crashing. Also, grep out your OOM's and check the timestamps to see if there is any regularity with the crashes over time. I have had to sleuth through some poorly crafted cron jobs in my time. I would make sure to have at least 2GB of swap if you have < 8GB of RAM. Like I said, swapping will slow things down, but it is better than crashing the DB and losing transactions and having to check/repair tables on start up.

Best Answer

Related Solutions

Linux OOM-kill why

Mysql crashing, oom-killer, out of memory, tuning issues

Related Topic