My test system setup is a simple raidz 5 disk (8 TBs each) zpool with 256 GB RAM. Performance starts out good, achieving around 650 MB/sec throughput. Eventually, the Linux Buffer cache fills up and starts to cause the ZFS ARC cache to shrink until it reaches around 32 MB in size. Once this occurs, throughput drops to around 60 MB/sec.
I have tried some kernel tuning options such as setting vm.vfs_cache_pressure to 200 but did not change the problem from occurring.
My current workaround is to set /sys/module/zfs/parameters/zfs_arc_min to 128 GB and zfs_arc_max to 252 GB. Because of the Linux buffer cache, the ARC never grows above 128 GB but performance stays at a throughput level of 650 MB/sec for the entire data transfer.
Ideally, I would like to be able to disable Linux buffer cache all together, and let the ZFS ARC cache be between 0 and 256 GB of memory, leaving the rest of the memory available to application work load. If it were possible to give the ZFS ARC cache higher priority than the Linux buffer cache, that would also be an acceptable answer as it would effectively disable the Linux buffer cache.
Thanks,
Zack
Edit 1:
The issue I am trying to solve is the Linux buffer cache evicting the ZFS ARC from memory. I would like the application workload to be able to use all of the memory, and the ZFS ARC shrink accordingly. As workaround, I had to set the ZFS ARC min size to a large value as the Linux buffer cache would evict the ARC from memory, if the min size was set to 0.
Best Answer
Set a minimum ARC value. That's all that's necessary.
The ARC will grow/shrink as needed. Whatever is eating buffet cache must not be on a ZFS file system. What is this server doing?