I've got an Ubuntu VM, running inside Ubuntu-based Xen XCP. It hosts a custom FCGI-based HTTP service, behind nginx
.
Under load from ab
the first CPU core is saturated, and the rest is under-loaded.
In /proc/interrupts
I see that CPU0 serves an order of magnitude more interrupts than any other core. Most of them come from eth1
.
Is there anything I can do to improve performance of this VM? Is there a way to balance interrupts more evenly?
Gory details:
$ uname -a Linux MYHOST 2.6.38-15-virtual #59-Ubuntu SMP Fri Apr 27 16:40:18 UTC 2012 i686 i686 i386 GNU/Linux $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 11.04 Release: 11.04 Codename: natty $ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 283: 113720624 0 0 0 0 0 0 0 xen-dyn-event eth1 284: 1 0 0 0 0 0 0 0 xen-dyn-event eth0 285: 2254 0 0 3873799 0 0 0 0 xen-dyn-event blkif 286: 23 0 0 0 0 0 0 0 xen-dyn-event hvc_console 287: 492 42 0 0 0 0 0 295324 xen-dyn-event xenbus 288: 0 0 0 0 0 0 0 222294 xen-percpu-ipi callfuncsingle7 289: 0 0 0 0 0 0 0 0 xen-percpu-virq debug7 290: 0 0 0 0 0 0 0 151302 xen-percpu-ipi callfunc7 291: 0 0 0 0 0 0 0 3236015 xen-percpu-ipi resched7 292: 0 0 0 0 0 0 0 60064 xen-percpu-ipi spinlock7 293: 0 0 0 0 0 0 0 12355510 xen-percpu-virq timer7 294: 0 0 0 0 0 0 803174 0 xen-percpu-ipi callfuncsingle6 295: 0 0 0 0 0 0 0 0 xen-percpu-virq debug6 296: 0 0 0 0 0 0 60027 0 xen-percpu-ipi callfunc6 297: 0 0 0 0 0 0 5374762 0 xen-percpu-ipi resched6 298: 0 0 0 0 0 0 64976 0 xen-percpu-ipi spinlock6 299: 0 0 0 0 0 0 15294870 0 xen-percpu-virq timer6 300: 0 0 0 0 0 264441 0 0 xen-percpu-ipi callfuncsingle5 301: 0 0 0 0 0 0 0 0 xen-percpu-virq debug5 302: 0 0 0 0 0 79324 0 0 xen-percpu-ipi callfunc5 303: 0 0 0 0 0 3468144 0 0 xen-percpu-ipi resched5 304: 0 0 0 0 0 66269 0 0 xen-percpu-ipi spinlock5 305: 0 0 0 0 0 12778464 0 0 xen-percpu-virq timer5 306: 0 0 0 0 844591 0 0 0 xen-percpu-ipi callfuncsingle4 307: 0 0 0 0 0 0 0 0 xen-percpu-virq debug4 308: 0 0 0 0 75293 0 0 0 xen-percpu-ipi callfunc4 309: 0 0 0 0 3482146 0 0 0 xen-percpu-ipi resched4 310: 0 0 0 0 79312 0 0 0 xen-percpu-ipi spinlock4 311: 0 0 0 0 21642424 0 0 0 xen-percpu-virq timer4 312: 0 0 0 449141 0 0 0 0 xen-percpu-ipi callfuncsingle3 313: 0 0 0 0 0 0 0 0 xen-percpu-virq debug3 314: 0 0 0 95405 0 0 0 0 xen-percpu-ipi callfunc3 315: 0 0 0 3802992 0 0 0 0 xen-percpu-ipi resched3 316: 0 0 0 76607 0 0 0 0 xen-percpu-ipi spinlock3 317: 0 0 0 16439729 0 0 0 0 xen-percpu-virq timer3 318: 0 0 876383 0 0 0 0 0 xen-percpu-ipi callfuncsingle2 319: 0 0 0 0 0 0 0 0 xen-percpu-virq debug2 320: 0 0 76416 0 0 0 0 0 xen-percpu-ipi callfunc2 321: 0 0 3422476 0 0 0 0 0 xen-percpu-ipi resched2 322: 0 0 69217 0 0 0 0 0 xen-percpu-ipi spinlock2 323: 0 0 10247182 0 0 0 0 0 xen-percpu-virq timer2 324: 0 393514 0 0 0 0 0 0 xen-percpu-ipi callfuncsingle1 325: 0 0 0 0 0 0 0 0 xen-percpu-virq debug1 326: 0 95773 0 0 0 0 0 0 xen-percpu-ipi callfunc1 327: 0 3551629 0 0 0 0 0 0 xen-percpu-ipi resched1 328: 0 77823 0 0 0 0 0 0 xen-percpu-ipi spinlock1 329: 0 13784021 0 0 0 0 0 0 xen-percpu-virq timer1 330: 730435 0 0 0 0 0 0 0 xen-percpu-ipi callfuncsingle0 331: 0 0 0 0 0 0 0 0 xen-percpu-virq debug0 332: 39649 0 0 0 0 0 0 0 xen-percpu-ipi callfunc0 333: 3607120 0 0 0 0 0 0 0 xen-percpu-ipi resched0 334: 348740 0 0 0 0 0 0 0 xen-percpu-ipi spinlock0 335: 89912004 0 0 0 0 0 0 0 xen-percpu-virq timer0 NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 0 0 0 0 0 0 0 0 Local timer interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts RES: 3607120 3551629 3422476 3802992 3482146 3468144 5374762 3236015 Rescheduling interrupts CAL: 770084 489287 952799 544546 919884 343765 863201 373596 Function call interrupts TLB: 0 0 0 0 0 0 0 0 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts MCE: 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 0 0 0 0 0 0 0 0 Machine check polls ERR: 0 MIS: 0
Best Answer
Look in the
/proc/irq/283
directory. There is asmp_affinity_list
file which shows which CPUs will get the 283 interrupt. For you this file probably contains "0" (andsmp_affinity
probably contains "1").You can write the CPU range to the
smp_affinity_list
file:Or you can write a bitmask, where each bit corresponds to a CPU, to
smp_affinity
:However, irqbalance is known to have its own idea of what affinity each interrupt should have, and it might revert your updates. So it is best if you just uninstall irqbalance completely. Or at least stop it and disable it from coming up on reboot.
If even without irqbalance you are getting odd
smp_affinity
for interrupt 283 after a reboot, you will have to manually update the CPU affinity in one of your startup scripts.