Linux – How to find which cgroup caused OOM

cgrouplinuxoom

I'm using cgroup to partition my processes and I'm getting Out Of Memory messages in my kernel logs.

However, I can't find which partition causes them. I've checked the memory controller cgroup but there are no obvious ways to use it.

The problem is, by the time I see 'task killed' message in system logs the task is dead, its /proc entry is gone and cgroup's tasks file doesn't have the pid of the killed task.

Best Answer

Answering my own question. I've used SystemTap to hook into the OOM killer:


#!/usr/bin/env stap
%{
#include <linux/cgroup.h>
%}

function find_mem_cgroup:string(task:long) %{
    struct cgroup *cgrp;
    struct task_struct *tsk = (struct task_struct *)((long)THIS->task);

    /* Initialize with an empty value */ 
    strcpy(THIS->__retvalue, "NULL");

    cgroup_lock();
    cgrp = task_cgroup(tsk, mem_cgroup_subsys_id);
    if (cgrp)
        cgroup_path(cgrp, THIS->__retvalue, MAXSTRINGLEN);
    cgroup_unlock();
%}

probe kernel.function("oom_kill_task") {
    cgroup = find_mem_cgroup($p)
    exename = kernel_string($p->comm)
    printf("pid\t%d\tmem-cgroup\t%s\texe-name\t%s\n", $p->pid, cgroup, exename)
}

Works like this:

cyberax@cybnb:~/work/shell$ sudo stap -g oom.stap
pid    3966    mem-cgroup    /task1/1/    exe-name    oom_generator.p
Related Topic