Linux – IOAT enabled but not being used with Linux

centosintellinuxnetworking

On our R610s with igb version 4.1.2 I do see the ioatdma module loading (and loaded with lsmod):

ioatdma: Intel(R) QuickData Technology Driver 4.00
ioatdma 0000:00:16.0: irq 80 for MSI/MSI-X
ioatdma 0000:00:16.1: irq 81 for MSI/MSI-X
ioatdma 0000:00:16.2: irq 82 for MSI/MSI-X
ioatdma 0000:00:16.3: irq 83 for MSI/MSI-X
ioatdma 0000:00:16.4: irq 84 for MSI/MSI-X
ioatdma 0000:00:16.5: irq 85 for MSI/MSI-X
ioatdma 0000:00:16.6: irq 86 for MSI/MSI-X
ioatdma 0000:00:16.7: irq 87 for MSI/MSI-X

We have a quad port Intel nic that should support it:

07:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

However, it looks like it isn't being used for any network traffic:

[root@or-rtlb01 ~]# cat /sys/class/dma/dma*/bytes*
0
0
0
0
0
0
0
0

Does anyone know how to make it so ioat is used for network traffic?

Kernel is Linux or-rtlb01.ds.stackexchange.com 3.7.7-1.el6.elrepo.x86_64 #1 SMP Mon Feb 11 22:28:07 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

Best Answer

How big are the packets you're sending through? By default the IOATDMA driver is only used on large packet sizes. You can view the current setting with:

cat /proc/sys/net/ipv4/tcp_dma_copybreak

If you want the value to be lower, you can set it using sysctl. For example, to set the packet size to 1, use:

sysctl net.ipv4.tcp_dma_copybreak=1