Assume the following drive setup (with ZFS):
Controller 1 Drive1 Drive4 Drive7 Drive10 Controller 2 Drive2 Drive5 Drive8 Drive11 Controller 3 Drive3 Drive6 Drive9 Drive12
VDEV setup:
vdev1: drive1, drive2, drive3
vdev2: drive4, drive5, drive6
vdev3: drive7, drive8, drive9
vdev4: drive10, drive11, drive12
Is it better (for reliability) to add all vdevs to the same zpool, or have separate zpools (one for each vdev)? Also, if we lost a single vdev, would we lose the entire array? We don't need all the storage capacity in one place – the smaller zpools would be find from the storage standpoint.
Update: For 3dinfluence's question about the boot pool, that will be on a RAID1 set. I don't like mixing the OS and my multi-terabyte RAID arrays.
Best Answer
The way ZFS works is that inside the pool it has raid sets or groups. To expand capacity you have to add additional groups at the same raid level to the pool. IO is then stripped across all the groups inside the zpool that have free blocks. So a zpool with a large number of small disk sets is fast and highly available.
So I would suggest that you put all the drives in one pool as either 3 disk raidz groups or 6 2 drive mirror groups.
Raidz option
Pros
Mirror option
Pros
Cons