On an AWS instance x1.32xlarge (128 cores), we are getting a lot of interrupts per seconds.
Here are the top CPUs in interrupts/s:
Interrupts Top CPUs
CPU0: 140838.0
CPU1: 77867.0
CPU4: 66495.0
CPU6: 59941.0
CPU3: 39096.0
CPU2: 31532.0
CPU7: 30861.0
CPU5: 26042.0
CPU8: 4168.0
CPU12: 3026.0
CPU10: 2793.0
Here are the top interrupts/s/CPU:
Interrupts above 10k/s
HYP [Hypervisor callback interrupts] [CPU0] = 46902.0/sec
49 [xen-percpu-ipi resched0] [CPU0] = 43437.0/sec
RES [Rescheduling interrupts] [CPU0] = 41512.0/sec
HYP [Hypervisor callback interrupts] [CPU2] = 26638.0/sec
HYP [Hypervisor callback interrupts] [CPU8] = 22875.0/sec
HYP [Hypervisor callback interrupts] [CPU12] = 20813.0/sec
55 [xen-percpu-ipi resched1] [CPU2] = 20749.0/sec
RES [Rescheduling interrupts] [CPU2] = 19568.0/sec
73 [xen-percpu-ipi resched4] [CPU8] = 16400.0/sec
RES [Rescheduling interrupts] [CPU8] = 15677.0/sec
HYP [Hypervisor callback interrupts] [CPU6] = 14226.0/sec
85 [xen-percpu-ipi resched6] [CPU12] = 14060.0/sec
RES [Rescheduling interrupts] [CPU12] = 13271.0/sec
HYP [Hypervisor callback interrupts] [CPU14] = 12173.0/sec
HYP [Hypervisor callback interrupts] [CPU4] = 11887.0/sec
HYP [Hypervisor callback interrupts] [CPU10] = 10500.0/sec
This happens when the application running on that machine is under significant load. The network traffic is relatively high, and there are lots of threads.
My question is: are 50K/150K interrupts/sec too much? How do we interpret that number? Is there a maximum interrupts/sec?
UPDATE:
Here here a glimpse at the top output:
Tasks: 825 total, 3 running, 822 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.6%us, 3.4%sy, 0.0%ni, 83.6%id, 0.0%wa, 0.0%hi, 2.3%si, 0.0%st
Mem: 2014742856k total, 40059184k used, 1974683672k free, 162036k buffers
Swap: 0k total, 0k used, 0k free, 3159112k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32936 ec2-user 20 0 77.3g 11g 29m S 1759.7 0.6 1780:36 java
32118 ec2-user 20 0 64.2g 10g 26m S 1036.9 0.6 62:31.08 java
3 root 20 0 0 0 0 R 70.4 0.0 14:54.84 ksoftirqd/0
12 root 20 0 0 0 0 S 21.2 0.0 6:06.47 ksoftirqd/1
16 root 20 0 0 0 0 S 15.2 0.0 4:33.28 ksoftirqd/2
20 root 20 0 0 0 0 S 12.2 0.0 3:34.12 ksoftirqd/3
28 root 20 0 0 0 0 S 11.9 0.0 3:24.96 ksoftirqd/5
24 root 20 0 0 0 0 S 11.6 0.0 3:26.54 ksoftirqd/4
32 root 20 0 0 0 0 S 10.2 0.0 3:23.56 ksoftirqd/6
36 root 20 0 0 0 0 S 10.2 0.0 3:28.80 ksoftirqd/7
UPDATE2:
Best Answer
Most of the interrupts were from net network cards queues, this allowed to spread the load onto others cores: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-cpu-irq.html