For a certain read/write workload, I get 30 MiB/s from a 4-disk array using esata port multiplication. When I move the exact same array to four individual sata connections, I get 90 MiB/s for the same workload. Is this the expected performance impact of the multiplier? If not, how do I narrow the problem down between the software, controller, or enclosure?
I've gotten the exact same results even after trying a large number of variables over the years:
- Many Linux versions (RHEL5, Fedora 9, 11, 13, 16) and all the kernels that entails.
- Two types of enclosures (both low-end $100 4-disk models), four enclosures in total.
- Two types SATA controller chipsets (Marvell 88SE91xx and Silicon Image 3132)
- Four SATA controllers (Around $30 – $60 pci-x1 interface cards and one PCI, all FIS-based switching).
- Three SATA speeds (1.5-, 3.0-, and 6.0-Gbps)
- Four types of hdds (Samsung Spinpoint F1 1TB, WD Caviar Black 2TB, Seagate Barracuda XT 3TB).
- Three computers (Athlon 3ghz+1GB RAM when I made the first e-sata array, then Core 2 Duo+4GB, now a Core i5 750+8GB).
When I did my first eSATA enclosure all those years ago, I assumed the slow speed was related to using a PCI (33mhz) card, immature SATA drivers, or perhaps the SATA-1.5Gbps 5400 RPM drives. But now with an x1 6Gbps controller, 3.0 kernel, and 6Gbps 7200rpm HDDS, still nothing has changed. I still get the same 30 MiB/s for that particular workload until I move it to individual SATA connections.
Is this a fundamental limitation of all port multiplication implementations?
Best Answer
Yes. Poor performance is to be expected.
The ATA protocol was never intended to carry data to more than one drive (it was actually based on the ISA Bus, extending the communication channel to a storage drive). The IDE interface cheated this limitation by carrying data for "Master" and "Slave" devices. SATA has no equivalent mechanism however, as it was intended for only a single device per cable connection.
eSATA Port Multipliers function as an ATA target that can select multiple physical media (so only one physical device can be addressed at a time, and there is a slight latency in switching to a different device). You could compare this to old Optical Libraries.
This is radically different from SAS which was built with Expanders and initiator/target routing built-in (SAS 2.0+ functions somewhat like a switched Ethernet network). SAS is also able to bond multiple lines into trunks allowing simultaneous access to multiple devices across the multiple trunk lines.
Update:
Apparently there are newer controllers that support FIS, which allows multiple drives to be "active" at the same time. This should allow performance much higher than described above. Apparently at this time only one manufacturer makes these chips. I would still highly recommend starting away from SATA in server environments. If you must use SATA drives consider still using SAS controllers and Expanders.