TL;DR: CPU-hungry processes are sharing cores on a multi-core server while other cores sit idle, whereas I expect the Linux scheduler to distribute jobs evenly across cores. What could cause this behavior?
My workgroup has a new 4-core server running SuSE that is experiencing some strange CPU-scheduling behavior that our admin (the guy who set up the box) didn't know about and can't seem to fix. I'm hoping you all can help me diagnose what might be causing this strange behavior. Here are the symptoms:
1) Every SSH session seems to be limited to using a single core. I have tested this a variety of ways, but the simplest was creating a simple infinite-looping C program and running multiple copies. They always share one core if started from a single SSH session, and I can't even control which core; it seems to be set at login time. Furthermore, even using multiple simultaneous SSH sessions, the only cores that I can ever utilize this way are cores 0 and 3; 1 and 2 never get touched, no matter how many sessions or processes I start.
2) If, from an SSH session like the ones mentioned above, I start my program with "nohup" to divorce it from the current session, it will use a different core than the rest of the programs started from that session. However, all nohup-started programs from the same SSH session will again share a core with each other. Interestingly, these nohup-started programs are always assigned to cores 1 and 2.
The expected behavior, of course, is what I've always seen on other Linux systems (I'm mostly familiar with RHEL, Fedora, and Ubuntu): I should be able to utilize all 4 cores from a single session with or without using nohup; furthermore, jobs should occasionally jump cores to balance the load time evenly between them.
Here's an example of 2 processes running on one core: http://i.imgur.com/K9rH3.png (Sorry, can't post images directly yet, even though I've got enough cred on other StackExchange sites). Note in the above that each "burn" process takes up 100% of one core if run in isolation, but here they are sharing a core for some reason while three cores sit idle. Also note that these two processes shared the "Cpu3" core in excess of 20 minutes without changing to another core to balance the load (this was after the image was taken; after 20 minutes I stopped watching).
My first thought when encountering this problem was that "ulimit" was in effect, but it doesn't look like it to my (admittedly non-expert) eye:
dmonner@bass:~> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128465
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 13988192
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 128465
virtual memory (kbytes, -v) 40007280
file locks (-x) unlimited
Additionally, /etc/security/limits.conf is empty (or at least has no lines that aren't comments), so I don't think it's PAM enforcing limits.
I have sudo access on the machine, and even if I 'sudo su' and then run processes, I am core-limited in the same way.
So: Does anyone have ideas as to what can cause this kind of behavior, and how to get rid of it?
Best Answer
The Linux scheduler will (if the application doesn't do it on it's own) never bind anything to any specific core. Quite the contrary actually, most of the time the scheduler will bounce the load across all cores at a seemingly random fashion. It is the only part of the kernel which has a kind of holistic view of the system and it tries to "optimize" system response/throughput according some heuristics which in term manifests itself as the kernel bouncing the processes across the cores.
You have to explicitly pin processes to certain cores via something like
taskset(1)to accomplish that. Can you provide some evidence for what you are experiencing?