Ubuntu – How to plan your hard drives in a ZFS pool

1. Offline drive #1.
2. Replace the old drive with the newer, high-capacity drive.
3. zpool replace the drive.
4. Wait for the replacement to be rebuilt.
5. Goto step 1 and repeat for each successive drive in the array.

You may need to export and then import the zpool once this procedure is complete; the pool will then show the increased size. No downtime is required, but if a drive does go south while you're upgrading your array, you will lose everything, as with four total disks you presumably have a 3+1 setup. (The odds aren't that high, but Murphy never sleeps.)

The official source: "Replacing Devices in a Storage Pool" in the ZFS Administration Guide. The ZFS Best Practices Guide is also useful.

If you're familiar with traditional RAID, basically you're comparing RAID6 (double parity) to RAID 50 (striped raid5 sets). The 8 disk RAID-z2 setup is probably the better bet as then any two of the 8 disks to fail without losing all your data, while with a striped pair of 4disk raidz1 sets can only tolerate one disk failure per set (failure of two disks in the same set and you loose everything).

The 2x4disk RAIDz1 sets may offer increased IOPS in some situations as smaller reads/writes might be serviced by one half or the other, whereas every IO hits every disk in the 8disk raidz2 setup, but if performance is your primary concern you should definitely think about mirroring your disks instead (ala RAID 10).