Freebsd – Slow read and write speeds on FreeBSD ZFS

freebsdraidstoragezfs

my home storage server, which currently consists of 6 x 3 TB Seagate drives, 12 GB DDR3 RAM and a Core i3 540, has really poor performance when reading from or writing to the raidz2 I set up on.

Using /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1024 count=10:

10240 bytes transferred in 0.000221 secs (46331902 bytes/sec)

and the other way around:

10240 bytes transferred in 0.000107 secs (95656287 bytes/sec)

which translates to 46 MB/s write and 95 MB/s read speed. This seems kinda slow. Or is this perfectly normal?

Thanks

Tobias Timpe

Best Answer

With the command shown above, you aren't testing disk write performance. You're testing an assortment of factors but mostly controller and disk latency. To test disk read performance, you need to jump through hoops to exclude disk caching from your tests, which is non-trivial with ZFS as it means disabling the ARC cache. It's much easier to test write performance: write much larger files, preferably sizes exceeding your systems build-in RAM.

I ran this command on my home ZFS file server (Core i3-4130T with mirrored WDC WD80):

# /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1024 count=10000000
10240000000 bytes transferred in 55 secs (186900359 bytes/sec)

It's not great but that is a much better test of actual write performance. An interesting result is that I achieved 187MB/s and my disks rated performance is 178MB/s. That's a bit odd b/c typical benchmarks are some high fraction of rated performance. Exceeding it can be explained by ZFS's disk compression.

My disks are in a ZFS mirror (2x read performance is more useful to me than storage efficiency) but assuming your disks are also 5400 RPM, you should expect comparable write performance.

To see why even a much larger count using dd isn't a great test, check this out:

# /usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1M count=1000000
1048576000000 bytes transferred in 223.56 secs (4690377542 bytes/sec)

Simply by increasing the blocksize to 1M, my disk performance is now off the charts amazing. Until you realize that reading from /dev/zero means the data source has no entropy and is highly compressible. dd is not the best tool for testing disk performance.

There are other tools in the FreeBSD ports tree (I've used bonnie & iozone in the past) which can tell you more information about your disks performance. If you're really curious, look into them. Otherwise, a MUCH better performance test for a home file server is to copy the contents of a DVD or blu-ray between two disks and time that copy.

Related Topic