ZFS – zpool ARC cache plus L2ARC benchmarking

hard drivesolarisssdvirtualizationzfs

I have been doing lots of I/O testing on a ZFS system I will eventually use to serve virtual machines. I thought I would try adding SSD's for use as cache to see how much faster I can get the read speed. I also have 24GB of RAM in the machine that acts as ARC. vol0 is 6.4TB and the cache disks are 60GB SSD's. The zvol is as follows: 

pool: vol0
 state: ONLINE
 scrub: none requested
config:

        NAME                     STATE     READ WRITE CKSUM
        vol0                     ONLINE       0     0     0
          c1t8d0                 ONLINE       0     0     0
        cache
          c3t5001517958D80533d0  ONLINE       0     0     0
          c3t5001517959092566d0  ONLINE       0     0     0

The issue is I'm not seeing any difference with the SSD's installed. I've tried bonnie++ benchmarks and some simple dd commands to write a file then read the file. I have run benchmarks before and after adding the SSD's.

I've ensured the file sizes are at least double my RAM so there is no way it can all get cached locally.

Am I missing something here? When am I going to see benefits of having all that cache? Am I simply not under these circumstances? Are the benchmark programs not good for testing the effect of cache because of the the way (and what) it writes and reads?

Best Answer

It seems your tests are very sequential like writing a large file with dd then reading it. ZFS L2ARC cache is designed to boost performance on random reads workloads, not for streaming like patterns. Also, to get optimal performance, you might want to wait a longer time until the cache is warm. Another point would be to make sure your working set fit into the SSDs. Having io statistics observed during the tests would help figuring out what devices are used and how they perform.

Related Topic