Slow iSCSI speeds between WinServer2016 and Win10 over 10G Ethernet

iscsinetwork-sharewindows 10windows-server-2016

I have set up an iSCSI share on the Win Server 2016 machine on a RAID0 array that has r/w speeds around 680/670MBps. The problem is that I've got a dynamic vhdx file on that array, connected to a single client, and when I'm trying to read or write on that drive, it tops out at 350MBps. Otherwise, when I'm reading directly from the array over the 10G connection, the speed goes up to 520MBps, which is the maximum speed of the SSD in the Win10 client.

I'm using Mellanox ConnectX-2 cards and 10GBASE-LR transceivers. Have jumbo frames, and everything else enabled, and I do get higher speeds if I'm not using that specific iSCSI drive. The server does not seem to struggle when I'm copying, I mean it has an 8-core Opteron and 16GB ram, so that should be enough.

Is there a way to make it faster, or is that the limit of iSCSI?

Thanks a lot!

Best Answer

The regular MSFT iSCSI target is very slow, that's a proven fact unfortunately. There are plenty other programs that have it's own iSCSI target.

I'm using Starwinds in our production environment, you can take a look at it. It also has a free version. HPE VSA are also good, but they're configured in virtual machines, so that's a little "-" to the performance.

Hope that helps.

Related Solutions

Nfs – What are “good” speeds for iSCSI and NFS over 1GB ethernet

We've pushed a Sun X4100 writing over bonded GigE and iSCSI to an Sun X4500 to 280MB/s.

There's a lot that can be done to tune the TCP stack on Solaris to help things out, this my my stock tuning config. (take from some collection of Sun whitepapers):

$ cat /etc/rc3.d/S99ndd
#!/bin/bash

NDD=/usr/sbin/ndd

$NDD -set /dev/tcp tcp_xmit_hiwat 1048576
$NDD -set /dev/tcp tcp_recv_hiwat 8388608
$NDD -set /dev/tcp tcp_max_buf 8388608
$NDD -set /dev/udp udp_xmit_hiwat 1048576
$NDD -set /dev/udp udp_recv_hiwat 8388608
$NDD -set /dev/udp udp_max_buf 8388608
$NDD -set /dev/tcp tcp_conn_req_max_q 65536
$NDD -set /dev/tcp tcp_conn_req_max_q0 65536
$NDD -set /dev/tcp tcp_fin_wait_2_flush_interval 67500

Also, worth looking into on your OpenSolaris machine is changing the fsflush interval, the interrupt adjustment "magic" and disabling soft rings. Append the following to /etc/system (reboot required):

* "fsflush" tuning
set tune_t_fsflushr = 5
set autoup = 300
* Disable the Automatic Interrupt Adjustment
set dld:dld_opt = 2
* Disable "soft rings"
set ip:ip_squeue_fanout = 0
set ip:ip_soft_rings_cnt = 0

Worth mentioning, I do this on Solaris 10 -- not OpenSolaris -- but I think the tunables should work for you just the same.

I'm a big fan of Filebench for playing around with tuning options and doing throughput tests.

The (recently renamed) OpenSolaris 2009.06 release looks to be very exciting in the world of iSCSI and ZFS.

Hope this helps some!

How to crack the 1 Gbit iSCSI limit between ESXi and VNXe

It is possible that you do not generate enough IOPS for this to really kick-in.
Have a look here on how to change the setting from default 1'000 IOPS to a smaller value. (This is symmetrix specific, but you can do the same for the VMWare Round Robin Provider)

I'm however not yet convinced if it really is able to utilize more than one link totally in parallel with just one datastore. I think you have to do the IOMeter test on more than one datastore in parallel to see benefits. (Not 100% sure though)

Best Answer

Related Solutions

Nfs – What are “good” speeds for iSCSI and NFS over 1GB ethernet

How to crack the 1 Gbit iSCSI limit between ESXi and VNXe

Related Topic