Yes, I have encountered this with low-end cards and buggy drivers. However, no, not on an up-to-date Adaptec rebranded card. Wow is all I can say. One thing to consider, maybe it is more a bug with the drive than the RAID controller.
I don't have a good answer, but since you seem to have exhausted most of your options other than replacing the card, (and replacing the drives did the trick) here's a few ideas you can consider for your troubleshooting:
The WD drives were RE (RAID Edition) drives, right? The time limited error recovery is important, so if you don't have that and the drive is attempting to recover the sector, you are going to get a looooong pause from that drive. If the RAID controller is being patient and not dropping the drive you'll have a big problem on your hands.
Check the SMART data on the drives you removed and see if there is anything interesting.
Another comment about the importance of time limited error recovery (TLER) feature, from NAS / RAID vendor support:
As I mention before, we always suggest customers to use enterprise level drives if they use the drives in RAID settings. Enterprise level drives have more consistent responding time so that the RAID will be safer.
Regarding Backplanes
It varies from vendor to vendor, but in general backplanes are not compatible with of the shelf hard-drives. Many need some kind of drive carrier that has the built in interface between the SAS connector and the backplane connector. This is because these kinds of systems are hot-plug, and that requires special bits.
Regarding RAID controllers
Hardware RAID provides a level of parallel processing that can come very much in handy, as well as handling certain tasks better than software RAID can. One area is the on-adapter cache, which allows the RAID card to better virtualize the underlaying storage so it performs better. Software RAID can do some of that, but hardware RAID still performs better these days. Also, in my experience HW RAID handles failures more gracefully than SW RAID. Yours may vary.
Regarding RAID and ZFS
This is going to sound a bit odd, but I run into the same issues with NetWare's NSS file-system (which looks a lot like ZFS as it happens). In my case I trust the hardware vendors more to handle complex storage configs than I trust the software vendors to provide solid solutions. This may be misplaced trust, but I'd rather have a storage management system with several largish RAID arrays, than one with 48 individual disk drives. This allows me to leverage the best of both environments.
I can go into some detail about load leveling on hardware and software, but that's a bit beyond the scope of this article ;)
Regarding attaching external SAS arrays
If I'm reading that SUN unit correctly, it's a JBOD unit by itself. Attach it to a SAS RAID controller with external ports and you can use hardware RAID on it. Or attach it to a stand alone SAS card and have up to 48 individual drives presented to the operating system. Either method will work. Whether or not the SAS RAID card can be configured for JBOD is up to the RAID card manufacturer, I've seen it go both ways over the years.
Regarding "4 (x4-wide) SAS host/uplink ports (48 Gb/sec bandwidth)"
This means that the unit has multiple SAS ports on it, and it can do link aggregation for increased bandwidth. To make full use of this, you'll need 4 free ports on the card you attach it to. These also can be used to attach two hosts to this unit, if you're of a mind.
The 'Expansion ports' on the spec are for attaching additional SAS shelves to the first unit. You'd attach your RAID card to the first unit, and then attach additional units to the first over those expansion ports. I think. Trough this you can get silly amounts of direct-attach-storage.
Regarding standard ports
Some of this varies from vendor to vendor, but in general 1U-2U servers these days do not ship with external storage connectors standard. The 4U servers may be different, but I don't play with those that often so I don't know first hand. To get the ability to use external storage, you'll need an adapter card of some kind. Whether that's a simple SAS adapter, or a smarter version of the built in RAID adapter is up to you.
Best Answer
2 They don't supply 2TB disks for it yet, but they likely will do and it will almost certainly support them
3 Yes, you can put in as many or as few disks, though obviously less than 5-6 it doesn't make much sense to be using it
4 The DS3400 is the identical array with a fibre channel interface, it's benchmarked using SPC-1 and SPC-2 here, the performance will be very similar
5 The server's SAS card sees the RAID controller in the DS3200 rather than the raw disks. The presentation is controlled via the IBM Storage Manager client software which connects over TCP/IP. You can build multiple RAID arrays on the DS3200, which would each appear as single disks to the server.
6 The full configuration guide is available as an IBM Redbook and includes multiple screenshots
7 The DS3200 internal controller handles the RAID
The IBM DS3000 series arrays are pretty good at what they do, they're also pretty dumb compared to most other arrays out there, but they are cheap. It's based on an LSI model, Dell sell an essentially identical MD3000 disk array.
Hope that all helps