Nfs – Disk throughput dramatically reduced when using ZFS on OpenSolaris

nfsopensolarisperformancezfs

I'm building a simple ZFS file server for the small business I work for. The server is a Dell Poweredge 840, with 1GB RAM. The OS (OpenSolaris 2009.06) is installed on one SATA drive, and there are three other SATA drives installed for storage: 1x1TB, 1×1.5TB, and 1x2TB. When I add the three drives to one raidz zpool, throughput isn't very good:

#zpool create -m /export/pool pool raidz c7d1 c8d0 c8d1
#zfs create pool/fs
#time dd if=/dev/zero of=/export/pool/fs/zerofile bs=1048576 count=1024

1024+0 records in
1024+0 records out

real    0m12.539s
user    0m0.002s
sys     0m0.435s

That's about 81.6 MB/s. That's not horrendous, but I tried creating a pool consisting of just one of those drives:

#zpool create -m /export/disk-c7d1 disk-c7d1 c7d1
#zfs create disk-c7d1/fs
#time dd if=/dev/zero of=/export/disk-c7d1/fs/zerofile bs=1048576 count=1024
1024+0 records in
1024+0 records out

real    0m21.251s
user    0m0.002s
sys     0m0.552s

Okay, 48.19 MB/s throughput for a sequential write to one drive? That seems pretty low. Especially when I format the drive as UFS and try that same write:

#newfs /dev/dsk/c7d1s2
<snip>
#mount /dev/dsk/c7d1s2 /mnt/c7d1
# time dd if=/dev/zero of=/mnt/c7d1/zeroes bs=1048576 count=1024
1024+0 records in
1024+0 records out

real    0m10.372s
user    0m0.002s
sys     0m1.720s

That's almost twice the speed, 98.73 MB/s. That's much closer to what I'd expect out of these drives (though they're just cheap SATA drives).

What am I doing wrong here? I understand that there's overhead involved in writing parity data with RAIDZ, but making a pool from a single drive shouldn't halve throughput, should it? That seems pretty bad.

Thanks, everybody.

Best Answer

ZFS performance can be greatly improved by adding read and write cache SSDs (i know i know they're pricey) Check this link for more info.