Most discussions of ZFS suggest that the hardware RAID be turned off and that ZFS should directly talk to the disks and manage the RAID on the host (instead of the RAID controller).
This makes sense on a computer with 2-16 or even more local disks, but what about in an environment with a large SAN?
For example, the enterprise I work for has what I would consider to be a modest sized SAN with 2 full racks of disks, which is something like 400 spindles. I've seen SAN shelves that are way more dense than ours, and SAN deployments way larger than ours.
Do people expose 100 disks directly to big ZFS servers? 300 disks? 3000 disks? Do the SAN management tools facilitate automated management of this sort of thing?
Best Answer
FWIW I have experience with up to 92 disks in a single ZFS pool and so far it works fine.
But if you're really talking about several hundreds of disks I would consider partitioning them into a small number of disjunct (but still large) pools. I don't want to know how long e.g. a
zpool scrubruns on a 3000 disk pool (but you want to scrub regularly). Also the output of commands likezpool statuswould be unwieldy with such a large number of disks. So why put all eggs into a single basket?(Side note on dedup: Notice that although dedup can be controlled at the dataset level it will find duplicates at the pool level. I.e. you'll probably get worse dedup results if you're partitioning as suggested. On the other hand you'll need much more memory to hold the dedup hashes of a single giant pool which might not fit into ARC+L2ARC if the pool's too big. So if you are using dedup the amount of available memory is probably a good indicator for the maximum practical pool size.)