Ubuntu – Low performance both iSCSI and AoE

iscsiperformanceUbuntu

We are looking for a resonable speed storage. Because of low budget we decided to use software iSCSI or AoE targets.
Before we change our production infrastucture, we are doing some tests to choose the best technology.

For testing we use:

  • Fujitsu Siemens RX200 S4 as target
  • Fujitsu Siemens RX200 S4 as initiator
  • NetGear managed 1GBit switch
  • onboard NICs (Broadcom w/TOE), EdiMax NICs, Broadcom NICs w/TOE – all 1GBit
  • target server is using a QLogic controller with 6 2TB WD Blue SATA drives.
  • both target and initiator operating systems are Ubuntu 16.04 LTS with all updates.
    Switch is dedicated for storage purpose. We test bonds and multipathing.

Our problem is low read speed. For testing we use dd and a 40-100GB file.

  • local read and write on a target server is over 300MB/s.
  • writing to server by iSCSI or AoE is over 200MB/s which satisfies us.
  • reading from server is always 95-99MB/s.

We have tried ietd, aoetools, LIO.
We have used bonds of 2 NICs: balance-rr and LACP, multipathing with r-r. Used normal and jumbo frames. Finally we even did direct ethernet connection between target and host (no switch).

All test give more less the same results (of course using common NICs without TOE and iSCSI gave 20-30% worse results).

Testing network with iperf showed transfers about 200MB/s (2GBit).
Watching NICs usage on target with bmon showed equal utilization of both devices (each about 50MB/s for reading, about 100MB/s for writing).

As we had no luck, we decided to use a third NIC (both sides of course). The results were strange:

  • 2 NICs – 50MB/s each
  • 3 NICs – 33MB/s each

Is there any limit on target software that disables output higher than 1GBit/s?

What do we do wrong?

Best Answer

To squeeze maximum performance out of iSCSI connected storage you should use Jumbo Frames and MPIO (not LACP). RDMA/iSER is recommended if you can do that.

AOE (ATA over Ethernet) is old and is shit. We’ve got rid of Coraid years ago, already. We are using StarWind https://www.starwindsoftware.com/ as iSCSI target quite a while already and StarWind asked us to migrate off Coraid to whatever storage we could do.

So right now, we are very good with iSCSI provided by StarWind and using Windows, ESX and SCST http://scst.sourceforge.net/ on Linux as initiators. With RDMA/iSER it does up to 10 Gbit, very happy so far.

Related Topic