Difference between LSI SAS3081E-R vs. LSI SAS3081E-S SAS/ATA HBAs

Nice, well prepared question :)

I'm a speeds'n'feeds-man myself and I think you're on the money to be honest. I was half expecting to see your throughput lower than it is but what I think you've got there is a build-up in minor, and expected, inefficiencies. For instance it's very hard for a PCIe bus to get to 100% all the time, better to assume a low 90% overall rate. Given the jitter this will cause it will also mean that the PHYs won't be 100% 'fed' all the time so you lose a bit there to, same for the cache, disks, non-coalraced interrupts, IO scheduling etc. Basically it minor ineffeciency times minor inefficiency times...and so on, it ends up being more than the 5-10% expected inefficencies on their own. I've seen this kind of thing with HP DL servers talking to their MSA SAS boxes using W2K3 and then being NLB'ed over multiple NICs - frustrating but understandable I guess. That's my 2c anyway, sorry it's not too positive.

This has been covered here... See the related links on the right pane of this question.

Right now, the market conditions are such that you should try to use SAS disks everywhere you can.

• Enterprise SAS disks are your fastest and most resilient rotating media available at 10,000 and 15,000 RPM. Performance-optimized
• Nearline or Midline SAS are usually mechanically-equivalent to 7,200 RPM SATA disks, but feature a SAS interface and offer the benefits of the SAS protocol. They are available in higher capacities than enterprise SAS disks. They have a slight price premium over the same sized SATA drives. Capacity-optimized
• Solid-state disks (SSDs) are a higher tier of storage and shouldn't be compared directly with rotating media. Their main characteristic is higher random read and write performance, but the details are beyond the scope of this question.

Also see:

SAS or SATA for 3 TB drives?

How can a single disk in a hardware SATA RAID-10 array bring the entire array to a screeching halt?