Well, I use a D2700 for ZFS storage and worked a bit to get LEDs and sesctl features to work on it. I also have SAS MPxIO multipath running well.
I've done quite a bit of SSD testing on ZFS and with this enclosure.
Here's the lowdown.
- The D2700 is a perfectly-fine JBOD for ZFS.
- You will want to have an HP Smart Array controller handy to update the enclosure firmware to the latest revision.
- LSI controllers are recommended here. I use a pair of LSI 9205-8e for this.
- I have a pile of HP drive caddies and have tested Intel, OCZ, OWC (sandforce), HP (Sandisk/Pliant), Pliant, STEC and Seagate SAS and SATA SSDs for ZFS use.
- I would reserve the D2700 for dual-ported 6G disks, assuming you will use multipathing. If not, you're possibly taking a bandwidth hit due to the oversubscription of the SAS link to the host.
- I tend to leave the SSDs meant for ZIL and L2arc inside of the storage head. Coupled with an LSI 9211-8i, it seems safer.
- The Intel and Sandforce-based SATA SSDs were fine in the chassis. No temperature probe issues or anything.
- The HP SAS SSDs (Sandisk/Pliant) require a deep queue that ZFS really can't take advantage of. They are not good pool or cache disks.
- STEC is great with LSI controllers and ZFS... except for price... They are also incompatible with Smart Array P410 controllers. Weird. I have an open ticket with STEC for that.
Which controllers are you using? I probably have detailed data for the combination you have.
Your system is definitely underperforming based on your hardware specifications. I loaded the sysbench
utility on a couple of idle HP ProLiant DL380 G6/G7 servers running CentOS 5/6 to check their performance. These are normal fixed partitions instead of LVM. (I don't typically use LVM, because of the flexibility offered by HP Smart Array controllers)
The DL380 G6 has a 6-disk RAID 1+0 array on a Smart Array P410 controller with 512MB of battery-backed cache. The DL380 G7 has a 2-disk enterprise SLC SSD array. The filesystems are XFS. I used the same sysbench command line as you did:
sysbench --init-rng=on --test=fileio --num-threads=16 --file-num=128 --file-block-size=4K --file-total-size=54G --file-test-mode=rndrd --file-fsync-freq=0 --file-fsync-end=off --max-requests=30000 run
My results were 1595 random reads-per-second across 6-disks.
On SSD, the result was 39047 random reads-per-second. Full results are at the end of this post...
As for your setup, the first thing that jumps out at me is the size of your test partition. You're nearly filling the 60GB partition with 54GB of test files. I'm not sure if ext4 has an issue performing at 90+%, but that's the quickest thing for you to modify and retest. (or use a smaller set of test data)
Even with LVM, there are some tuning options available on this controller/disk setup. Checking the read-ahead and changing the I/O scheduler setting from the default cfq to deadline or noop is helpful. Please see the question and answers at: Linux - real-world hardware RAID controller tuning (scsi and cciss)
What is your RAID controller cache ratio? I typically use a 75%/25% write/read balance. This should be a quick test. The 6-disk array completed in 18 seconds. Yours took over 2 minutes.
Can you run a bonnie++ or iozone test on the partition/array in question? It would be helpful to see if there are any other bottlenecks on the system. I wasn't familiar with sysbench, but I think these other tools will give you a better overview of the system's capabilities.
Filesystem mount options may make a small difference, but I think the problem could be deeper than that...
hpacucli output...
Smart Array P410i in Slot 0 (Embedded) (sn: 50123456789ABCDE)
array A (SAS, Unused Space: 0 MB)
logicaldrive 1 (838.1 GB, RAID 1+0, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 300 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 300 GB, OK)
SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 50123456789ABCED)
sysbench DL380 G6 6-disk results...
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.
Operations performed: 30001 Read, 0 Write, 0 Other = 30001 Total
Read 117.19Mb Written 0b Total transferred 117.19Mb (6.2292Mb/sec)
1594.67 Requests/sec executed
Test execution summary:
total time: 18.8133s
total number of events: 30001
total time taken by event execution: 300.7545
per-request statistics:
min: 0.00ms
avg: 10.02ms
max: 277.41ms
approx. 95 percentile: 25.58ms
Threads fairness:
events (avg/stddev): 1875.0625/41.46
execution time (avg/stddev): 18.7972/0.01
sysbench DL380 G7 SSD results...
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.
Operations performed: 30038 Read, 0 Write, 0 Other = 30038 Total
Read 117.34Mb Written 0b Total transferred 117.34Mb (152.53Mb/sec)
39046.89 Requests/sec executed
Test execution summary:
total time: 0.7693s
total number of events: 30038
total time taken by event execution: 12.2631
per-request statistics:
min: 0.00ms
avg: 0.41ms
max: 1.89ms
approx. 95 percentile: 0.57ms
Threads fairness:
events (avg/stddev): 1877.3750/15.59
execution time (avg/stddev): 0.7664/0.00
Best Answer
Your performance sounds about right. The P812 is a 6Gb card, and you're getting consistent 4Gb performance in a RAID10 configuration. Pretty strong, especially with only one enclosure and one pair of SAS channels in use. It shows both channels are actually being used, otherwise your performance would be closer to 375MB/s.
In order to get more performance, you're going to need more than one D2700 enclosure, and run them off of the second pair of ports on the P812 card. Set each enclosure up as a RAID0 LUN and then mirror them. That way, your mirroring I/O won't contend with each other and your throughput should increase significantly. You may not get much past 1GB/s though
The bigger question is what kind of I/O are you expecting this system to handle? You say "lots of I/O performance", but there are a couple of ways to define that. You seem to be focusing on simple throughput, but your choice of disks suggests latency is actually a major concern.
I'd suggest characterizing your storage performance across a variety of access sizes to get a better feel for its overall performance. If you know what kinds of disk transfers your high I/O application needs, focus especially on those ranges. Also pay attention to stripe size on the RAID sets and where your partition-breaks fall. This is more typically an SSD concern, but you seem to want max-possible performance so the extra percentage points you get for ensuring your filesystem blocks align with the RAID stripes is worth looking in to.
Simple file-copy is not enough to characterize the performance of a storage system. For that you need real benchmarks. I'm particularly fond of IOZone (link), but IOMeter (link) has better market penetration. Focus your testing on data-sizes you're likely to be working with and I/O transfer sizes you're likely to use. It can be very amazing how different storage performs when working with 4KB chunks and 32KB chunks.