SAS Expander backplane performance drops when more than 8 drives are active

hardwaresasstoragesupermicro

We are using an LSI 9211-8i raid controller (flashed in IT mode) in a PCI-e 2.0 x8 slot.
It's connected to a supermicro SAS 836EL1 backplane with 1 SFF-8087 cable.
We have 16 drives installed.

When we do a 'badblocks' test of one disk performance is about 140 mbyte/s.
When doing badblocks tests concurrently on more drives all drives keep getting about 140mbyte/s until we exceed 8 drives.

With 16 drives active we only get 60mbyte/s on each disk.

Is this normal behavior for an expander backplane? If not, what could be the cause?

Best Answer

This is how SAS expanders work. You're technically oversubscribed, but in reality, you shouldn't be.

  • Your WD2003FYYS disks link at 3.0Gbps, but are only capable of 1.2Gbps/disk.
  • Your Supermicro drive backplane/expander has 16 ports with 6.0Gbps downlinks to the drives, but a single SFF-8087 uplink to the controller.
  • From what I can tell, this backplane only supports a single SAS SFF-8087 connection to an HBA. The 836EL2 model supports multiple connections.

enter image description here

The SAS SFF-8087 connection is capable of 4 x 6.0Gbps == 24Gbps throughput.

140 Megbytes/second is ~1.2Gbps. 8 disks at that rate is about 9Gbps. You can probably get a few more disks going before rates drop. Account for some overhead.

You're topping out at 3.0Gbps speeds (12Gbps ceiling on a 4-lane SAS cable).

There may be a link negotiation issue between your LSI HBA and the expander backplane. Check the LSI controller and see if it is showing 3.0Gbps link on the SFF-8087 port or 6.0Gbps. You're only getting 3.0Gbps speeds, but all of the devices in the chain (except for the disks) are 6.0Gbps.

Have you brought the firmware of the expander up-to-date?

Edit:

I think I ran into this at work... There was an issue with this expander type and LSI controllers that caused SATA drives to link at 1.5Gbps. Take a look here. So the negotiation issues may be on the drive side. Can you also check firmware on your LSI controller.

Related Topic