I originally posted on "Network Engineering", but I was told to ask here.
I have 2 servers connected with 10G cable between them directly. I need to know that the connection is good and it has indeed 10G throughput. If not how I can check what is the issue? I used IPERF to check the bandwidth.
UPDATE
I configured jumbo packets, extreme interrupt rate, max values for receive/transmit buffers.
For the IPERF I used just "iperf3.exe -s" for server and "iperf3.exe -c IP" for client.
According to the test the bandwidth is 1.5-1.7 Gbits/sec. Does it mean the connection is 10G? What could be the problem?
Server A is Windows 8.1 Pro 64bit
Server B is Windows 7 Enterprise 64bit
This is the output when ServerA (192.168.10.100) is server and ServerB (192.168.10.200) is client:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.200, port 63772
[ 5] local 192.168.10.100 port 5201 connected to 192.168.10.200 port 63773
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 170 MBytes 1.42 Gbits/sec
[ 5] 1.00-2.00 sec 217 MBytes 1.82 Gbits/sec
[ 5] 2.00-3.00 sec 217 MBytes 1.82 Gbits/sec
[ 5] 3.00-4.00 sec 214 MBytes 1.80 Gbits/sec
[ 5] 4.00-5.00 sec 217 MBytes 1.82 Gbits/sec
[ 5] 5.00-6.00 sec 207 MBytes 1.74 Gbits/sec
[ 5] 6.00-7.00 sec 199 MBytes 1.67 Gbits/sec
[ 5] 7.00-8.00 sec 206 MBytes 1.73 Gbits/sec
[ 5] 8.00-9.00 sec 192 MBytes 1.61 Gbits/sec
[ 5] 9.00-10.00 sec 209 MBytes 1.75 Gbits/sec
[ 5] 10.00-10.16 sec 30.3 MBytes 1.63 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.16 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.16 sec 2.03 GBytes 1.72 Gbits/sec receiver
This is the output when B (192.168.10.200) is server and A (192.168.10.100) is client:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.100, port 55565
[ 5] local 192.168.10.200 port 5201 connected to 192.168.10.100 port 55566
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 170 MBytes 1.43 Gbits/sec
[ 5] 1.00-2.00 sec 175 MBytes 1.47 Gbits/sec
[ 5] 2.00-3.00 sec 175 MBytes 1.47 Gbits/sec
[ 5] 3.00-4.00 sec 179 MBytes 1.50 Gbits/sec
[ 5] 4.00-5.00 sec 181 MBytes 1.52 Gbits/sec
[ 5] 5.00-6.00 sec 200 MBytes 1.68 Gbits/sec
[ 5] 6.00-7.00 sec 176 MBytes 1.47 Gbits/sec
[ 5] 7.00-8.00 sec 177 MBytes 1.48 Gbits/sec
[ 5] 8.00-9.00 sec 186 MBytes 1.56 Gbits/sec
[ 5] 9.00-10.00 sec 179 MBytes 1.50 Gbits/sec
[ 5] 10.00-10.04 sec 6.42 MBytes 1.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.04 sec 1.76 GBytes 1.51 Gbits/sec receiver
UPDATE2 – UDP
When 192.168.10.100 is server
iperf3.exe -c 192.168.10.100 -u -V
iperf 3.1.3
CYGWIN_NT-6.1 ServerB 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Tue, 09 Aug 2016 19:58:03 GMT
Connecting to host 192.168.10.100, port 5201
Cookie: ServerB.1470772683.561800.7c902a3e559
[ 4] local 192.168.10.200 port 63675 connected to 192.168.10.100 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 1
0 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 4.00-5.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 8.00-9.00 sec 128 KBytes 1.05 Mbits/sec 16
[ 4] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 16
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datag
rams
[ 4] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.437 ms 0/160 (0%)
[ 4] Sent 160 datagrams
CPU Utilization: local/sender 1.7% (0.9%u/0.8%s), remote/receiver 0.1% (0.0%u/0.
1%s)
When 192.168.10.200 is server
iperf3.exe -c 192.168.10.200 -u -V
iperf 3.1.3
CYGWIN_NT-6.3 ServerA 2.5.1(0.297/5/3) 2016-04-21 22:14 x86_64
Time: Tue, 09 Aug 2016 20:37:11 GMT
Connecting to host 192.168.10.200, port 5201
Cookie: ServerA.1470775031.986727.6f8e913650af9b
[ 4] local 192.168.10.100 port 51834 connected to 192.168.10.200 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 1
0 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 152 KBytes 1.24 Mbits/sec 19
[ 4] 1.00-2.00 sec 144 KBytes 1.18 Mbits/sec 18
[ 4] 2.00-3.00 sec 160 KBytes 1.31 Mbits/sec 20
[ 4] 3.00-4.00 sec 160 KBytes 1.31 Mbits/sec 20
[ 4] 4.00-5.00 sec 152 KBytes 1.25 Mbits/sec 19
[ 4] 5.00-6.00 sec 152 KBytes 1.24 Mbits/sec 19
[ 4] 6.00-7.00 sec 152 KBytes 1.25 Mbits/sec 19
[ 4] 7.00-8.00 sec 152 KBytes 1.25 Mbits/sec 19
[ 4] 8.00-9.00 sec 152 KBytes 1.24 Mbits/sec 19
[ 4] 9.00-10.00 sec 144 KBytes 1.18 Mbits/sec 18
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datag
rams
[ 4] 0.00-10.00 sec 1.48 MBytes 1.25 Mbits/sec 0.202 ms 0/189 (0%)
[ 4] Sent 189 datagrams
CPU Utilization: local/sender 0.1% (0.1%u/0.0%s), remote/receiver 0.0% (0.0%u/0.
0%s)
UPDATE3 – NTTTCP W/O -NDL
ServerA
ntttcp.exe -s -m 8,*,192.168.10.200 -l 128k -a 2 -t 15
Copyright Version 5.31
Network activity progressing...
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 15.000 65544.533 131072.000
1 15.000 65518.933 131072.000
2 15.000 93585.067 131072.000
3 15.002 63906.146 131072.000
4 15.000 65536.000 131072.000
5 15.000 62310.400 131072.000
6 15.001 81837.744 131072.000
7 15.001 73262.316 131072.000
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
8371.875000 15.001 8854.867 558.088
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
4464.702 17.198 66975.000
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
12120.992 0.823 30450.170 0.328
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
991381 149616 9 0 36.026
ServerB
ntttcp.exe -r -m 8,*,192.168.10.200 -rb 2M -a 16 -t 15
Copyright Version 5.31
Network activity progressing...
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 15.000 65546.450 60268.990
1 14.999 65523.318 60229.282
2 14.999 93582.889 60659.943
3 14.999 63900.843 61621.870
4 15.000 65536.817 60278.174
5 14.999 62316.071 61141.628
6 14.999 81840.156 60394.032
7 14.999 73264.934 60858.538
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
8371.320061 14.999 8942.835 558.125
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
8930.003 5.314 133941.121
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
7965.264 8.216 14260.151 4.589
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
146841 981564 1 0 9.699
UPDATE4 – NTTTCP WITH -NDL
ServerA
ntttcp.exe -s -m 8,*,192.168.10.200 -l 128k -a 2 -t 15 -ndl
Copyright Version 5.31
Network activity progressing...
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 14.999 78059.604 131072.000
1 15.000 76902.400 131072.000
2 14.999 59882.392 131072.000
3 14.999 62485.232 131072.000
4 14.999 72734.449 131072.000
5 15.001 77520.165 131072.000
6 15.000 62702.933 131072.000
7 15.000 62771.200 131072.000
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
8101.250000 15.000 8912.094 540.083
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
4320.667 17.355 64810.000
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
9966.333 0.974 29515.333 0.329
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
953174 145579 1 0 35.182
ServerB
ntttcp.exe -r -m 8,*,192.168.10.200 -rb 2M -a 16 -t 15 -ndl
Copyright Version 5.31
Network activity progressing...
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 15.000 78054.000 60255.789
1 15.001 76903.606 60839.375
2 15.000 59880.183 61786.888
3 14.999 62483.832 60812.856
4 15.000 72730.333 61516.405
5 14.999 77520.335 60398.365
6 14.998 62701.660 60628.917
7 14.998 62771.436 60585.766
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
8100.944093 14.999 8943.556 540.099
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
8641.583 4.888 129615.105
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
7912.594 8.003 14043.670 4.509
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
143914 949785 0 0 8.633
Best Answer
It turn out that the problem was related to how Windows 7/8 clients manage the TCP Window scaling.
By default, Windows client editions prefer relatively small TCP window, prioritizing latency vs throughput. When using very fast link, as in this case, setting auto-tuning to normal can bring significant bandwidth gains.
To set auto-tuning to saner value, you can use the command
netsh interface tcp set heuristics disabled
. Anyway, give a look here for more information.