Linux – how to check which CPU cores are serving rx queues for a particular NIC and if RPS is enabled

linuxnetworking

On my host (RHEL 6.3, 2.6.x kernel) , eth2 generates irq # 60-65. I validated it via /proc/interrupts.

The /proc/irq/61-65/smp_affinity shows following bitsets for 5 queues

  1. 1000

  2. 0040

  3. 0400

  4. 0100

  5. 0010

I am reading them as CPU Core Ids:-

  1. 12 (for bitmask 1000)
  2. 6 (for bitmask 0040)
  3. 10 (for bitmask 0400)
  4. 2 (for bitmask 0004)
  5. 8 (for bitmask 0100)
  6. 4 (for bitmask 0010)

Am i reading them correct?

I am running a network application, which receives 10k messages per second on eth2. Oddly enough, i dont see any of the above CPU Core Ids busy, rather only odd numbered cpu core ids are busy. I am unable to co-relate smp_affinity with cpu activity.

Also, /sys/class/net/eth2/queues/rx-[0-4]/rps_cpus is 0000. Does it mean RPS is disabled?

UPDATE:1 In addition to above, irqbalance daemon is running. Can that override smp_affinity?

UPDATE:2 irqbalance –debug shows following

package 0: containing cpu 0, 2, 4, 6, 8, 10, 12
package 1: containing cpu 1, 3, 5, 7, 9, 11, 13, 15

How does irqbalance uses package information for numa based systems. running lstopo on my system yields
Socket 0
Node 0 : Cores 0, 2, 4, 6
Node 1: Cores 8, 10, 12, 14

Socket 1
Node 2: Core 1, 3, 5, 7
Node 3: Core 9, 11, 13, 15

Looking at numa topology and packages in irqbalance, it seems like irqs will be spread across Nodes, but limited to a Socket. Not sure why.

Also, using lspci, i can see eth2 is using pci bus 4 and Node 0 is on pci bus 4. So most normal thing in my mind, would be to spread eth 2 interrupts over Node 0, rather it seems to be spread over Socket 1.

Best Answer

I am reading them as CPU Core Ids:- … Am i reading them correct?

Since basically those numbers are just powers of 2 for each proc number, you can use this snippet which uses bc to do log2 calculation to get N from powers:

for N in 1000 40 400 100 10; do
    echo -n "$N is "
    echo "ibase=16; l($N) / l(2)" | bc -l | sed -e 's!\.000.*!!g'
done

→

1000 is 12
40 is 6
400 is 10
100 is 8
10 is 4

UPDATE: In addition to above, irqbalance daemon is running. Can that override smp_affinity?

Sure, in case you want manual control you'd need to disable irqbalance.

Related Topic