To be able to increase storage space by replacing only a few of the disks, you should use mirrored vdevs, striped together (which amounts to RAID10 indeed).
In your case, with 4 drives that would mean working to something like this:
zpool
mirror
disk1
disk2
mirror
disk3
disk4
This would provide you with 2TB of storage (given all disks are 1TB) and good redundancy (0% of array failure after 1 disk crash, only 33% of array failure with 2 simultaneous disk crashes).
Now to get there I would buy those 2 new 1TB disks, and put those in the pool:
zpool create zpool mirror disk1 disk2
Then move your stuff of the DLINK to the newly created pool.
Once that is done, you can scavenge the DLINK disks and add them to the pool, to increase storage:
zpool add zpool mirror disk3 disk4
If you later want to increase storage even more, you can do that by adding more vdevs (preferably also mirrors) OR by replacing only 2 of the 4 disks. Replacing goes as follows:
zpool offline zpool disk3
# remove physical disk3 at this point
# insert new, bigger disk in place of disk3
zpool online zpool disk3
# wait for resilver
# after resilver, do the same with disk4
# your vdev is now bigger, increasing the size of the pool
Now, let's look at the other option. If you had made 1 raidz vdev like so:
zpool
raidz
disk1
disk2
disk3
disk4
You would have 3TB of storage, but, to increase that storage by just replacing disks (and not adding), you would have to replace ALL 4 disks (one by one ofcourse) to increase pool size! Also this configuration has 100% array failure if 2 disks crash simultaneously.
The raidz configuration would also by slower than the striped mirrors configuration. Since raidz is more computationally intensive, while the stripes + mirrors actually improve read and write performance.
With 'normal' harddisks (non SSD) the striped mirrors will likely fill your gigabit connection for sequential reads and writes, because ZFS can combine the disks' bandwidth (remember 1Gb/s is only ~125 MegaBYTES/s, a standard 'normal' harddisk will give you around 90 Megabytes/s). I don't think the above raidz configuration will be able to do that on consumer hardware.
To conclude, the score for striped mirrors / RAID 10 with your amount of disks is:
+ max redundancy
+ maintenance
- available storage space
+ speed
The score for raidz is:
- max redundancy
- maintenance
+ available storage space
- speed
I would say striped mirrors win :)
A final tip: definitely read up more on the how-to and the why before starting! Maybe even simulate the whole procedure in a Virtual Machine. I'm thinking particularly on the step where you add the second mirror vdev! If you do it wrong you might get a different configuration that you had hoped for and ZFS is very unforgiving in those cases, since it doesn't allow you to remove vdevs from the pool or disks from raidz vdevs!! (removing disks form mirror vdevs is allowed however)
Also, be future proof and label and align your disks, so you don't get into trouble with Advanced Format drives! For more information on the intricacies of ZFS and 4K drives, I suggest you read this thread on the FreeBSD forum.
Ugly, but this would work.
Except when it doesen't;).
- Be very careful when specifying the partitions and when replacing the disks
- try it in am VM beforehand, setup the virtual disks like your hardware an dry run it 1 or 2 times.
- make a scrub before you start and take a look at the S.M.A.R.T info from the disks. You would not try this with an already flakey disk.
Important: You better have a tested backup on another medium or machine before trying it!
Yes, ZFS will grow the pool if the last 2TB disk or partition is replaced with a 4TB one ( if you have autoexpand=on for the pool )
zpool get autoexpand $pool
zpool set autoexpand=on $pool
On a sidenote: you should not use RAID-Z on disks bigger than 2TB. Your chance of getting an error on resilvering when replacing a faulted disk is very high. Please consider RAID-Z2.
Best Answer
These paragraphs are misleading and misinformed.
ZFS doesn't automatically add a newly inserted disk to a pool. You have to run the
zpoolcommand to tell how you want the disk to be used.There is mirroring only if Ari asked for but given the remaining of the sentence, that can't be the case.
If mirroring has been chosen, there can be no striping. If striping has been chosen, there can be no mirroring. In the latter case, there is no "striping in the background" or "post configuration striping" with ZFS. Striping is done once at write time.
Maybe, but easier than with traditional volume management.
Triple mirroring will give both the best read performance and the best security here.
You cannot set properties on directories, you do it with datasets (e.g. file systems) and pools. Creating multiple file systems is a lightweight operation with ZFS though.
To summarize:
You cannot have a mirrored file system and a striped one in a single pool. All datasets belonging to a pool share the same underlying pool RAID configuration.
Assuming ditto blocks (e.g.
zfs set copies=2 dataset) are what is used to provide fault tolerance, ZFS will do its best but there is no 100% guarantee all block pairs will end up to a different disks.Even in the case all ditto blocks are physically located on different disks, the pool would not survive a whole disk failure (or more precisely, would not survive a reboot after a whole disk failure). ZFS copies will protect against limited media errors (see Andreas paragraph) but not a whole disk unavailability. Calling Ari configuration "mirroring" is misleading.
I would never advice such a configuration. Ditto blocks are a great feature if you have a single disk but still want some level of protection against partial media failure (bad blocks) or if you use deduplication on a non redundant pool. Otherwise, expectations shouldn't be set too high and either mirroring or raidz are the way to go with multiple disks pools.
Note that a combination of mirroring, raidz and striping with three different size disks is doable with ZFS by creating multiple partition based pools. For example, let say you have a 4TB, a 3TB and a 2TB disks. You can split all your disks in 1TB partitions, create a 2 TB raidz with 1+1+1 partitions, a 3 TB stripe with 1+1+1 partitions, a 1 TB mirror with 1+1 partition and a 1 TB single device pool with the remaining partition. Do not expect stellar performance with such a mixed raid setup though and strive for the applications not to access concurrently the various pools.