Really low iops on ssd raid (lower than 1000)

hp-proliantiopsssdubuntu-14.04

I'm having serious performances problems on a server, with really low iops.

server :

  • ProLiant DL360p Gen8
  • with HP Smart Array P420i Controller (firmware v6)
  • with 4x Samsung 840 – 512GB on raid 1+0

here's one samsung 840 details :

hpacucli ctrl slot=0 pd 2I:1:6 show detail
physicaldrive 2I:1:6
Port: 2I
Box: 1
Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: Solid State SATA
Size: 512.1 GB
Firmware Revision: DXM04B0Q
Serial Number: S12SNEAD102607E
Model: ATA     Samsung SSD 840
SATA NCQ Capable: True
SATA NCQ Enabled: True
Current Temperature (C): 16
Maximum Temperature (C): 70
Usage remaining: 100.00%
Power On Hours: 0
SSD Smart Trip Wearout: False
PHY Count: 1
PHY Transfer Rate: 6.0Gbps
Drive Authentication Status: OK
Carrier Application Version: 11
Carrier Bootloader Version: 6

and here's the benchmark result

sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
    test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
    fio-2.1.3
    Starting 1 process
    test: Laying out IO file(s) (1 file(s) / 4096MB)
    Jobs: 1 (f=1): [m] [99.8% done] [7152KB/2400KB/0KB /s] [1788/600/0 iops] [eta 00m:02s]
    test: (groupid=0, jobs=1): err= 0: pid=36718: Thu Mar  5 18:15:12 2015
    read : io=3071.7MB, bw=2536.5KB/s, iops=634, runt=1240097msec
    write: io=1024.4MB, bw=866133B/s, iops=211, runt=1240097msec
    cpu          : usr=0.28%, sys=1.18%, ctx=401767, majf=0, minf=2347
    IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
    issued    : total=r=786347/w=262229/d=0, short=r=0/w=0/d=0
    Run status group 0 (all jobs):
    READ: io=3071.7MB, aggrb=2536KB/s, minb=2536KB/s, maxb=2536KB/s, mint=1240097msec, maxt=1240097msec
    WRITE: io=1024.4MB, aggrb=845KB/s, minb=845KB/s, maxb=845KB/s, mint=1240097msec, maxt=1240097msec
    Disk stats (read/write):
    dm-1: ios=785968/267543, merge=0/0, ticks=50476776/28341176, in_queue=79704028, util=100.00%, aggrios=788659/265218, aggrmerge=1953/2589, aggrticks=50709304/27483028, aggrin_queue=78191444, aggrutil=100.00%
    sda: ios=788659/265218, merge=1953/2589, ticks=50709304/27483028, in_queue=78191444, util=100.00%

running the same bench on normal hdd shows up more iops than on the ssd

Any idea ?

Best Answer

Due to how MLC SSDs work, they need a decently-sized local DRAM cache to absorb incoming writes while simultaneously write to the backing NAND.

However, hardware RAID cards often disable the disk's cache and exclusively rely on own (on-card) DRAM cache. While this is not a problem with classic HDD, the intrinsic not-overwriting nature of NAND chips (and SSDs build over them) cause a massive performance loss if the internal DRAM cache can not be used.

To give an example: a 256 GB Crucial M550, with sequential write of over 400 MB/s, drops to 5 MB/s with internal cache disabled. With random write the loss is equally severe. This is the precise reason why enterprise SSD used (in the past) SLC NANDs: they program time is way lower compared to MLC or even eMLC.

For this reason, I am always a little nervous when combining consumer-class SSDs with branded hardware RAID cards. On the other side, Linux mdraid works fantastically well with SSDs of any class.

For you specific case, you can do some test trying to re-enable the internal disk cache (DELL/LSI controller give this possibility, I don't know about HP controller). However, remember that with disk's write-cache enabled you are exposed to data loss in the event of a power outage (unless your disks have power-loss protection, but this is quite rare in the consumer space).

Related Topic