ZFS on Linux – Best Mountpoint Option for Manual Script Mounting

Congratulations and uh-oh. You've stumbled across one of the better things about ZFS, but also committed a configuration sin.

First, since you are using raidz1, you only have one disk worth of parity data. However, you had two drives fail contemporaneously. The only possible result here is data loss. No amount of resilvering is going to fix that.

Your spares helped you out a little bit here and saved you from a completely catastrophic failure. I'm going to go out on a limb here and say that the two drives that failed did not fail at the same time and that the first spare only partially resilvered before the second drive failed.

That seems hard to follow. Here's a picture:

This is actually a good thing because if this were a traditional RAID array, your entire array would have simply gone offline as soon as the second drive failed and you would have NO chance of an in-place recovery. But because this is ZFS, it can still run using the pieces it has and simply returns block or file level errors for the pieces it doesn't.

Here is how you fix it: Short-term, get a list of damaged files from zpool status -v and copy those files from backup to their original locations. Or delete the files. This will allow the resilver to resume and complete.

Here is your configuration sin: you have way too many drives in a raidz group.

Long term: you need to reconfigure your drives. A more appropriate configuration would be to arrange the drives in to small groups of 5 drives or so in raidz1. ZFS will automatically stripe across those small groups. This significantly reduces the resilver time when a drives fails because only 5 drives need to participate instead of all of them. The command to do this would be something like:

zpool create tank raidz da0 da1 da2 da3 da4 \
                  raidz da5 da6 da7 da8 da9 \
                  raidz da10 da11 da12 da13 da14 \
                  spare da15 spare da16

access denied on the child from the parent mount

sounds very much like what is explained in https://blog.programster.org/sharing-zfs-datasets-via-nfs (emphasis from original):

it is very easy to split your "pool" into as many datasets as you like. Each dataset is treated like its own filesystem, with its own rules and settings, which means with regards to sharing over NFS, that you can share more securely as client's will not be able to reach out of the bounds of that dataset/filesystem that you decided to share.

An important thing to bear in mind is that you will need to run the steps below for sharing on every pool/dataset that you wish to share, no matter the hierarchy. For example, if you have a dataset that is the parent of another, you would still have to set up sharing on that child if you wanted it to be shared, even if you have already set up sharing on the parent.

This might also be interesting for other people, who come here from a search engine.