I'm planning on building out a file server using ZFS and BSD, and I was hoping to make it more expandable by attaching drives stored in other machines in the same rack via iSCSI (e.g., one machine is running ZFS, and others have iSCSI targets available to be connected to by the ZFS box and added to zpools).
Looking for other people who have tried this has pretty much lead me to resources about exposing iSCSI shares on top of ZFS, but nothing about the reverse. Primarily I have the following questions:
- Is iSCSI over gigabit ethernet fast enough for this purpose, or would I have to switch to 10GbE to get decent performance?
- What would happen when one of the machines running iSCSI targets disconnects from the network?
- Is there a better way to do this that I just am not clever enough to have realized?
Capacity wise, initially this would be about 10TB of data (not accounting for redundancy), and a reasonable goal for the foreseeable future would be scaling to 20TB, so with redundancy, probably about 40-50TB of total storage. If possible, we also want to keep all the data encrypted using GELI.
Thanks for any help.
Best Answer
What is your goal in terms of capacity? This is definitely possible, as ZFS would be able to see your targets and aggregate them into pools. However, you're leaving a good amount of performance and reliability on the table.
My recommendation for scaling (at what I assume to be your capacity needs) is to invest in an external SAS multipath-cabale drive enclosure and ZFS-friendly controllers. If this is a situation where you need more than say, the 24TB of usable RAID 1+0 storage provided by one external enclosure filled with 2TB disks, then you're in the realm where you would benefit from expert design advice. At that level, iSCSI using disks present in other systems won't make things any faster.
The iSCSI design will fall over for latency, reliability and supportability reasons.