We have 20 2TB SATA drives to be used in a ZFS pool. I am after some advice on the best way to achieve good I/O performance, whilst being able to offer some redundancy (3 disk failures before data loss is what we are looking to achieve).
I am bit confused as to whether I need to use mirroring or raidz.
The 20 drives will be plugged into 2 16-port raid controllers (10 on each controller). Maybe I create hardware raid volumes for each lot of 10 disks and then in zfs, then mirror the two available raid volumes, thus creating one super volume?
Any advice would be great.
Best Answer
With 20 disks you have a lot of options. I'm assuming you already have drives for the OS, so the 20 disks would be dedicated data drives. In my Sun Fire x4540 (48 drives), I've allocated 20 drives in a mirrored setup and 24 in a striped raidz1 config (6 disks per raidz and 4 striped vdevs). Two disks are for the OS and the remainder are spares.
Which controller are you using? You may want to refer to: ZFS SAS/SATA controller recommendations
Don't use the hardware raid if you can. ZFS thrives when drives are presented as raw disks to the OS.
Your raidz1 performance increases with the number of stripes across raidz1 groups. With 20 disks, you could use 4 raidz1 groups consisting of 5 disks each, or 5 groups of 4 disks. Performance on the latter will be better. Your fault tolerance in that setup would be sustaining the failure of 1 disk per group (e.g., potentially 4 or 5 disks could fail under the right conditions).
The read speed from a raidz1 or raidz2 group is equivalent to the read speed of one disk. With the above setup, your theoretical max read speeds would be equivalent to that of 4 or 5 disks (for each vdev/group of raidz1 disks).
Going with the mirrored setup would maximize speed, but you will run into the bandwidth limitations of your controller at that point. You may not need that type of speed, so I'd suggest a combination of raidz1 and stripes. In that case, you could sustain one failed disk per mirrored pair (e.g. 10 disks could possibly fail if they're the right ones).
Either way, you should consider a hot-spare arrangement no matter which solution you go with. Perhaps 18 disks in a mirrored arrangement with 2 hot-spares or a 3-stripe 6-disk raidz1 with 2 hot-spares...
When I built my first ZFS setup, I used this note from Sun to help understand RAID level performance...
http://blogs.oracle.com/relling/entry/zfs_raid_recommendations_space_performance
Examples with 20 disks:
20-disk mirrored pairs.
20-disk striped raidz1 consisting of 4 stripes of 5-disk raidz1 vdevs.
Edit: Or if you want two pools of storage, you could break your 20 disks into two groups:
That gives you both types of storage, good redundancy, a spare drive, and you can test the performance of each pool back-to-back.