Set up simple Infiniband Block Storage (SRP or iSER)

hpcinfinibandstorage

I'm trying to figure out how to set up a simple storage system which exports block storage over Infiniband, using either SRP or iSER.

I'm very early in the process, and at the moment I'm basically just looking for a tutorial on the level of, "You have two servers on your fabric: here's how to export a block storage device from one to the other." Preferably on RHEL or variants thereof. Any pointers?

Best Answer

Well, to be frank I went the simple route and happily used iSCSI over IP over IB and it worked easily and performed well:

Infiniband IP setup ultra-quick primer.

first...

  • install opensm, infiniband-diags, rds-tools, sdpnetstat, srptools, perftest (for benchmarks)
  • load IB driver module, ib_umad, ib_ipoib
  • now you have a new network interface to configure.

performance settings:

  • connected mode, set the MTU to 65520
  • datagram mode, set the MTU to 2044
  • datagram performance : ~ 5 Gb/s
  • connected mode performance : ~ 6.3 Gb/s

YMMV with the IB controller model, driver, etc.

IP Settings :

net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.core.netdev_max_backlog=250000
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216
net.core.optmem_max=16777216
net.ipv4.tcp_mem="16777216 16777216 16777216"
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 65536 16777216"

Some documentations :

http://support.systemfabricworks.com/lxr/#ofed+OFED-1.5/ofed-docs-1.5/ipoib_release_notes.txt

http://www.mjmwired.net/kernel/Documentation/infiniband/ipoib.txt

iperf, 4 threads :

[  3] local 192.168.1.2 port 50585 connected with 192.168.1.3 port 5003
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.75 GBytes  2.36 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.79 GBytes  2.40 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.31 GBytes  2.84 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.51 GBytes  3.02 Gbits/sec

Total aggregated bandwidth 1.3 GB/s, definitely better than 10 GigE.

Related Topic