ZFS hot spares versus more parity

raidzfs

I am a bit new to ZFS but I am bringing up a new storage pool with 12 physical drives.

My current plan is to have RAIDZ2 across 10 drives and two hot spares.

But I wonder if I would not be better off with RAIDZ3 across all 12 drives and no hot spares.

The reasoning is that hot spares are basically powered-up, idle drives. It may be months or years before they are called into service and at that time I might find out that they were not viable. If they were part of a RAID stripe I could at least get feedback as to whether they were good or not.

I haven't seen much discussion of this on the web. Does anyone have advice?

Best Answer

About hot spares

Hot spares are set to a specific pool, but can be attached to any vdev of the pool automatically in case of failures. If you only have a single vdev consisting of all your disks, you are better off directly incorporating the disks (except if you already have RAIDZ3 and still disks to spare).

Additionally, resilvering takes time and occurs in a vulnerable (RAIDZ1, 2-way mirrors) or performance-reduced state (RAIDZ2, RAIDZ3, 3-way mirrors), which would not have occurred if you had already attached the device to the vdev.

Basically hot spares are a thing for large arrays. If you have 27 disks split into 3 vdevs of 9 disks in RAIDZ3, you can add 3 hot spares to reduce the "It's 2 AM and 3 disks have crashed, now I have to get up and fix this mess" moments (assuming a 32 drive bay system). Smaller systems usually don't have enough disks to even get to the "2+ vdevs and Z2/Z3" situation. An exception would be mirrors (e. g. 6 x 2), where crashes are much closer to being fatal for the pool (and you don't have enough disks to make them 6 x 3).

Optimal pool layout

Some advice from Nex7's blog regarding pool layout:

Do not use raidz1 for disks 1TB or greater in size.

For raidz1, do not use less than 3 disks, nor more than 7 disks in each vdev (and again, they should be under 1 TB in size, preferably under 750 GB in size) (5 is a typical average).

For raidz2, do not use less than 6 disks, nor more than 10 disks in each vdev (8 is a typical average).

For raidz3, do not use less than 7 disks, nor more than 15 disks in each vdev (13 & 15 are typical average).

Mirrors trump raidz almost every time. Far higher IOPS potential from a mirror pool than any raidz pool, given equal number of drives. Only downside is redundancy - raidz2/3 are safer, but much slower. Only way that doesn't trade off performance for safety is 3-way mirrors, but it sacrifices a ton of space (but I have seen customers do this - if your environment demands it, the cost may be worth it).

For >= 3TB size disks, 3-way mirrors begin to become more and more compelling.

This means in your case you would have the following options:

9 disks usable: (Z3 with 9+3)
8 disks usable: (Z2 with 4+2) ++ (Z2 with 4+2)
5 disks usable: (2-mirrors) * 5 ++ (hot spare) * 2
4 disks usable: (3-mirrors) * 4

I would rank them (descending) as:

In terms of usable space: 1, 2, 3, 4
In terms of safety: 1, 2/4, 3
In terms of speed: 4, 3, 2, 1
In terms of ability to extend/add drives: 3, 4, 2, 1

I would not use RAIDZ1 regardless of size, because you might want to later replace them with larger disks and then the problems will show (meaning you would not wnat to upgrade this way and might not be able to grow the storage space without adding additional disks).

Related Solutions

Freebsd – ZFS on FreeBSD: recovery from data corruption

The problem was that the new motherboard's BIOS created a host protected area (HPA) on some of the drives, a small section used by OEMs for system recovery purposes, usually located at the end of the harddrive.

ZFS maintains 4 labels with partition meta information and the HPA prevents ZFS from seeing the upper two.

Solution: Boot Linux, use hdparm to inspect and remove the HPA. Be very careful, this can easily destroy your data for good. Consult the article and the hdparm man page (parameter -N) for details.

The problem did not only occur with the new motherboard, I had a similar issue when connecting the drives to an SAS controller card. The solution is the same.

Debian – ZFS endless resilvering

Congratulations and uh-oh. You've stumbled across one of the better things about ZFS, but also committed a configuration sin.

First, since you are using raidz1, you only have one disk worth of parity data. However, you had two drives fail contemporaneously. The only possible result here is data loss. No amount of resilvering is going to fix that.

Your spares helped you out a little bit here and saved you from a completely catastrophic failure. I'm going to go out on a limb here and say that the two drives that failed did not fail at the same time and that the first spare only partially resilvered before the second drive failed.

That seems hard to follow. Here's a picture:

sequence of events

This is actually a good thing because if this were a traditional RAID array, your entire array would have simply gone offline as soon as the second drive failed and you would have NO chance of an in-place recovery. But because this is ZFS, it can still run using the pieces it has and simply returns block or file level errors for the pieces it doesn't.

Here is how you fix it: Short-term, get a list of damaged files from zpool status -v and copy those files from backup to their original locations. Or delete the files. This will allow the resilver to resume and complete.

Here is your configuration sin: you have way too many drives in a raidz group.

Long term: you need to reconfigure your drives. A more appropriate configuration would be to arrange the drives in to small groups of 5 drives or so in raidz1. ZFS will automatically stripe across those small groups. This significantly reduces the resilver time when a drives fails because only 5 drives need to participate instead of all of them. The command to do this would be something like:

zpool create tank raidz da0 da1 da2 da3 da4 \
                  raidz da5 da6 da7 da8 da9 \
                  raidz da10 da11 da12 da13 da14 \
                  spare da15 spare da16