Linux – isolcpus – binding not working

cpu-usagelinux

I am using isolcpus to isolate cores. I would like to bind specific threads to cores, but it is not working. The threads are moved to different cores after I bind them.

Cores 13, 14, and 15 are isolated:

$ cat /proc/cmdline
ro root=/dev/mapper/vg0-root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg0/swaprd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=137M@0M rd_NO_DM  KEYBOARDTYPE=pc KEYTABLE=us rd_LVM_LV=vg0/root rhgb quiet audit=0 intel_idle.max_cstate=0 console=tty0 console=ttyS1,115200 printk.time=1 processor.max_cstate=1 idle=poll biosdevname=0  isolcpus=13-15

top -H -p pgrep -u prusr12 Ser -d 1 shows this: 5017 and 5018 should have been bound to 14 and 15 and 5014 and 5016 should have been on 13.

PID  USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+   P COMMAND
5017 prusr12   20   0 1312m 1.1g 1.1g R 99.9  0.9   9:53.93  5 Server-3.10.
5018 prusr12   20   0 1312m 1.1g 1.1g R 99.9  0.9  10:08.88  7 Server-3.10.
5014 prusr12   20   0 1312m 1.1g 1.1g S  0.0  0.9   0:00.40  2 Server-3.10.
5016 prusr12   20   0 1312m 1.1g 1.1g S  0.0  0.9   0:01.04  4 Server-3.10.

The command line is this:

sg devuser "taskset -c 13 /releases/3.10.0/bin/Server-3.10.0 -n X -e DEV -p DEFAULT > /logs/ServerDevPR_DEFAULT.out 2>&1 &"

There are 4 threads in the process. I want the main thread to start on 13, hence taskset -c 13. Then two threads are spawed and will bind them to 14 and 15. I see that the threads were bound to 14 and 15, but then they were moved to other cores. pthread_setaffinity_np() is being used to bind the threads to cores.

Log after I bind the threads to 14 and 15:
CpuSet returned by pthread_getaffinity_np() contained:CPU 14
CpuSet returned by pthread_getaffinity_np() contained:CPU 15

System details:

$ uname -a
Linux host123 2.6.32-573.12.1.el6.x86_64 #1 SMP Mon Nov 23 12:55:32 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    1
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Stepping:              2
CPU MHz:               3199.847
BogoMIPS:              6399.06
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0-7
NUMA node1 CPU(s):     8-15

What could be going wrong? Thanks for your time.

Best Answer

Since I've worked with CPU isolation in the past, I created a tool to help me since I found out the hard way that it was surprisingly complex to get it right. The tool is available here:

https://github.com/OpenEneaLinux/rt-tools

It's the partrt script that you should look at and get inspired by.

One caveat though: It might not work well together with systemd out of the box, I have simply not tested this.

But as for your specific question, my guess is that what you need is to set /sys/fs/cgroup/cpuset/cpuset.sched_load_balance to 0, to disallow SMP load balancing. Now this will disable this for everything, which might not be what you want.

My tool (partrt) tries to divide the CPUs into two sets: One set which you want to work as your used to and the other set which tries to isolate each CPU within the set.

Related Topic