ZFS with L2ARC (SSD) slower for random seeks than without L2ARC

cacheopensolarisperformancessdzfs

I am currently testing ZFS (Opensolaris 2009.06) in an older fileserver to evaluate its use for our needs. Our current setup is as follows:

  • Dual core (2,4 GHz) with 4 GB RAM
  • 3x SATA controller with 11 HDDs (250 GB) and one SSD (OCZ Vertex 2 100 GB)

We want to evaluate the use of a L2ARC, so the current ZPOOL is:

$ zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        afstank      ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c11t0d0  ONLINE       0     0     0
            c11t1d0  ONLINE       0     0     0
            c11t2d0  ONLINE       0     0     0
            c11t3d0  ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c13t0d0  ONLINE       0     0     0
            c13t1d0  ONLINE       0     0     0
            c13t2d0  ONLINE       0     0     0
            c13t3d0  ONLINE       0     0     0
        cache
          c14t3d0    ONLINE       0     0     0

where c14t3d0 is the SSD (of course). We run IO tests with bonnie++ 1.03d, size is set to 200 GB (-s 200g) so that the test sample will never be completely in ARC/L2ARC. The results without SSD are (average values over several runs which show no differences)

write_chr       write_blk       rewrite         read_chr        read_blk        random seeks
101.998 kB/s    214.258 kB/s    96.673 kB/s     77.702 kB/s     254.695 kB/s    900 /s

With SSD it becomes interesting. My assumption was that the results should be in worst case at least the same. While write/read/rewrite rates are not different, the random seek rate differs significantly between individual bonnie++ runs (between 188 /s and 1333 /s so far), average is 548 +- 200 /s, so below the value w/o SSD.

So, my questions are mainly:

  1. Why do the random seek rates differ so much? If the seeks are really random, they should not differ much (my assumption). So, even if the SSD is impairing the performance it should be the same in each bonnie++ run.
  2. Why is the random seek performance worse in most of the bonnie++ runs? I would assume that some part of the bonnie++ data is in the L2ARC and random seeks on this data performs better while random seeks on other data just performs similarly like before.

Best Answer

Not sure why you're seeing the behavior you're seeing, but I can tell you why they don't necessarily signal terrible real world ZFS performance. Bonnie is designed to measure performance of the actual disks and intentionally tries to not leverage disk/memory cache. You're trying to use it to measure disk cache.

It can take hours for an L2ARC device to get hot. Depending on how much main memory you have for ARC and the disk performance with your workload, a 100GB SSD for L2ARC will take 1-2hrs to get hot and perhaps even longer (Source). Secondly L2Arc is designed to cache random reads and not streaming read workloads. "If you use the L2ARC for a streaming or sequential workload, then the L2ARC will mostly ignore it and not cache it" (Source). I would be really surprised if much of your bonnie workload ever made it into L2ARC. Instead of using bonnie++, try to generate load which resembles your real use of the system.

Although unlikely, running the latest development versions of Bonnie++ (1.96 vs 1.03d) may produce something closer to your expected results. You may also want to check out this sun blogs post about bonnie++ with the Sun 7000 Series, though they run over NFS and not locally.

Related Topic