zfs – ZFS Performance: Keep Free Space in Pool or File System?

quotasolariszfs

I know that the performance of ZFS heavily depends on the amount of free space:

Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. […] Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. ZFS_Best_Practices_Guide, solarisinternals.com (archive.org)

Now, suppose I have a raidz2 pool of 10T hosting a ZFS file system volume. Now I create a child file system volume/test and give it a reservation of 5T.

Then I mount both file systems per NFS to some host and perform some work. I understand that I can't write to volume more than 5T, because the remaining 5T are reserved to volume/test.

My first question is, how will the performance drop, if I fill my volume mount point with ~5T? Will it drop, because there is no free space in that file system for ZFS' copy-on-write and other meta-stuff? Or will it remain the same, since ZFS can use the free space within the space reserved for volume/test?

Now the second question. Does it make a difference, if I change the setup as follows? volume now has two file systems, volume/test1 and volume/test2. Both are given a 3T reservation each (but no quotas). Assume now, I write 7T to test1. Will the performance for both file systems be the same, or will it be different for every file system? Will it drop, or remain the same?

Thanks!

Best Answer

Yes. You need to keep free space in your pool. It's mainly for copy-on-write actions and snapshots. Performance declines at about 85% utilization. You can go higher, but there's a definite impact.

Don't mess with reservations. Especially with NFS. It's not necessary. Maybe for a zvol, but not NFS.

I don't see the confusion, though. If you have 10T, Don't use more than 85% of it. Size your shares appropriately, using quotas to cap their use. Or don't use any quotas and monitor your overall pool usage.