Will adding a SSD cache device to the ZFS storage improve performance

cacheiossdzfs

The server has 4GB of RAM and my zpool is made up of 15.5k SAS drives arranged like this:

    NAME          STATE     READ WRITE CKSUM
    tank          ONLINE       0     0     0
      raidz1-0    ONLINE       0     0     0
        c0t2d0    ONLINE       0     0     0
        c0t3d0    ONLINE       0     0     0
        c0t4d0    ONLINE       0     0     0
        c0t5d0    ONLINE       0     0     0
        c0t6d0    ONLINE       0     0     0
        c0t7d0    ONLINE       0     0     0
        c0t8d0    ONLINE       0     0     0
      raidz1-1    ONLINE       0     0     0
        c0t10d0   ONLINE       0     0     0
        c0t11d0   ONLINE       0     0     0
        c0t12d0   ONLINE       0     0     0
        c0t13d0   ONLINE       0     0     0
        c0t14d0   ONLINE       0     0     0
    spares
      c0t9d0      AVAIL
      c0t1d0      AVAIL

The primary use is as an NFS store for a couple VMWare ESXi servers. I can't do any "true" benchmarks because this is a production system (no budget for test systems), but using dd and bonnie++ I can't get more than ~40-50MB/s writes and ~70-90MB/s reads. It seems I should be able to do much better, but I'm not sure where to optimize. Based on what I've read, I think dropping in a OCZ Vertex 2 Pro SSD as my L2ARC is going to be the best bang-for-the-buck to improve througput.

  1. Is there something else I should be looking into to help performance? If not…
  2. How do I know how big a cache device I need?
  3. Am I safe with only a single SSD as my cache device?

Best Answer

My guess is that the ZFS Intent Log (ZIL) is limiting the write performance. A fast SSD used as a log device could help. Some hints:

  • ZIL Accelerator: DRAM or Flash? explains the issues in detail.
  • You may want to test with a disabled ZIL. See my other posting.
  • Checkout ZFS Features in Oracle Solaris Express if you're using a recent ZFS release because the way how to disable the ZIL has changed. There's a new sync filesystem property now.
  • Until recently it was impossible to remove a log device from a ZFS pool. Of course, this caused problems if the single log devices failed. It was therefore recommended to use a mirror for the log device. However, recent ZFS releases fixed this limitation and it's now possible to remove the single log device with zpool remove. Make sure you use a version that supports this feature if you really going to install the SSD.
Related Topic