NAS4Free disk encryption and ZFS error

This should work:

• Create a similar partition layout on the new disks, ZFS isn't going to do it for you.

• Copy the boot partition and reinstall the boot loader.

• set the autoexpand property on your root pool zpool set autoexpand=on labpool

• Replace one of the disks, eg zpool replace labpool sda5 sdc5 and wait for the resilvering to mirror all the pool datasets using zpool status

• Replace the second disk zpool replace labpool sdb5 sdd5.

• Remove the old disks

As ewwhite said, pool shrinking isn't currently possible with ZFS. If you need to do it, you'll have to backup to another storage medium (another pool, tape, SAN, etc) create a new pool and restore. As for expanding there are a number of options of how to grow your ZFS 5x3TB raidz pool:

1. Add a mirror VDEV (pairs of disks) pool spans the two VDEVs (12TB raidz & 3TB mirror).
2. Add a raidz VDEV (3-8 disks) pool spans the two VDEVs (12TB raidz & 12TB raidz).
3. Upgrade each disk (5x3TB to 5x4TB disks one at a time) pool on a single VDEV (16TB raidz).

Mixing multiple sizes or types of VDEVs (raidz + mirror) is non-optimal performance wise and offers the effective redundancy of the least redundant VDEV (raidz). At home or in a pinch you probably don't care, but it should be avoided if possible.

Realistically though, you shouldn't upgrade this pool. You shouldn't rely on RAID-Z and should switch to mirrors or RAID-Z2. It will cost you an extra disk (RAID-Z2) or two (mirroring) over RAID-Z to reach the same usable capacity, but there's a reason enterprises don't use RAID5 anymore. If a single disk fails, even with a hot spare ready to go it takes forever to recreate the missing disk from parity. Likely 24-48hrs or longer. If a second disk fails during this window (which is not unlikely because you're working the remaining disks as hard as you ever will, reading every byte off every disk as fast as you can) you will loose everything. With RAID-Z2 (think RAID6) two disks can fail without risking data loss. Consider the simplicity of mirrored VDEVs, but if cost/GB is your primary concern (at the expense of performance and expandability) double parity RAID-Z2 really is much safer.

That said, if you can find the temporary space and can afford the downtime, here are some more optimal ways to utilize your seven 3TB disks:

1. 6x3TB mirror + hotspare (9TB usable, 3 vdevs)
2. 6x3TB raidz2 set + hotspare (12TB usable, 1 vdev)
3. 7x3TB raidz2 set (15TB usable, 1 vdev)

If you'd chosen mirroring over raidz in the beginning, the initial five disks would've yielded a 4x3TB mirror + hotspare (6TB usable, 2 vdevs) and you could've just added pairs of disks as you needed them. Also, in case you're curious write performance is directly correlated with the number of VDEVs in the pool.

Really, it's all about how valuable your data is. If you have everything backed up elsewhere, you don't care about the 5% chance of a second disk failing during rebuild. But if you, like most people considering ZFS for a home server, have come to the conclusion that you won't (or can't afford to) perform regular backups and likely won't notice/replace a failed disk immediately, you should really consider the incremental cost of a hotspare and mirroring/RAID-Z2 instead of simple parity (RAID-Z) it just increases the odds you'll still have your data a few years down the road. ZFS goes to great lengths to use software to keep from loosing your data (checksums, parity/redundant copies, scrubbing, etc) without the need for an expensive and proprietary controller card. The least you can do is give ZFS the means means to protect against failed/failing disks: more than just the minimum number of disks required.