Nice, ionice are not enough

ionicenice

I have a script that starts big, CPU and memory consuming tree of processes. There is Python and executables down there, but everything starts with single bash script and python subprocesses.

During the execution, rest of system is completely chocked down. I attempted to do mitigate by
$ nice -n10 ionice -c2 ./Script.sh, however this is not sufficient – using computer is very laggy
(acutally this is development desktop, but the problem on designated server will be the same analogous).

I suspect, that the problem is with processes using to much memory – everything ends up swapped out and becomes sluggish.

Is there a way to lower priority of a process (and its recursive children) in access to physial memory?
I prefer it to be done slower in background with limited influence on other tasks.

Best Answer

You cannot limit "pace" of consuming memory, but you can limit it's total memory usage via various different mechanisms.

1) security limits Limit memory usage for user running the process via /etc/security/limits.conf. This may not work in your case if you are running this process as the same user working on different stuff.

Example:

username hard as 1000000

2) Control Groups You can - with cgroups, create a group and also limit the memory usage. Just create cgroup, like this:

# cat >> /etc/cgconfig.conf << EOF
group memlimit {
    memory {
        memory.limit_in_bytes = 1073741824;
    }
}
EOF

# cat >> /etc/cgrules.conf <<EOF
username memory   memlimit/
EOF

Offcourse - in both cases, you have to develop your program so that it can recover from failing to allocate more memory.

In case it can't, you just need to add more memory to your system, so that you can avoid swapping. Once the swapping starts, its in the hands of kernel, and you can't - for example - lower the priority of kswapd, and even if you could - that doesn't guarantee some of the programs you use wouldn't still get swapped out, thus causing even more slower system response. Just don't go there.

Related Topic