Slow SSD performance – IBM X3650 M4 (7915)

I set up a test environment for development purposes. It consists of a IBM X3650 M4 (7915) server with:

• 2 x Intel Xeon E2690 @ 2.90GHz
• 96GB 1333MHz ECC RAM
• 2 x HDD 146GB 15k rpm
• 6 x SSD 525GB (Crucial MX300)
• embedded ServeRaid m5110e in JBOD mode with no cache
• Ubuntu Server 16.10
• md RAID software on HDDs (RAID0) and SSDs (RAID10)

I can't totally bypass the RAID controller because it's integrated on the motherboard and don't have a dedicated HBA card (should I buy one?), but I set it to JBOD mode.

I tested and over-tested those SSDs as single disks, in RAID10 and RAID0 configurations. I observed expected behaviours from software RAID but not from single disks: RAID scales (it's ok for me) but single SSDs runs at half the expected IOPS!

Tests were run using fio and the configurations depicted by storagereviews.com ( Link ).

Here is a summary graph of averaged runs over all the 6 SSDs (1 x 60 seconds run on each SSD):

Reading from various benchmarks (storagereview.com, tomshardware.com, etc.) and from official specs, those disks should reach double random read IOPS. For example:

• for 4k workload tom's hardware tops at 92358 IOPS reads at 32 IO Depth while mine tops at ~37400 IOPS ( Link ).
• storagereview.com runs slightly different benchmarks but all of them give totally different results – ~90k IOPS for 4k aligned reads ( Link ).
• Hardware.info gives identical results for the 1TB model (Link).

I optimized yet all the various /sys/block/sd* and /dev/sd* parameters like scheduler, nr_requests, rotational, fifo_batch, etc.

What should I look for?

UPDATE 1

I forgot to mention that disks where overprovisioned at 25%, so overall size reported in the following outputs is roughly 75% of 525GB. Anyway, IOPS before and after overprovisioning never overpassed the 37k limit.

Output of hdparm -I /dev/sdc:

/dev/sdc:

ATA device, with non-removable media
    Model Number:       Crucial_CT525MX300SSD1                  
    Serial Number:      163113837E16
    Firmware Revision:  M0CR031
    Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
    Used: unknown (minor revision code 0x006d) 
    Supported: 10 9 8 7 6 5 
    Likely used: 10
Configuration:
    Logical     max current
    cylinders   16383   16383
    heads       16  16
    sectors/track   63  63
    --
    CHS current addressable sectors:   16514064
    LBA    user addressable sectors:  268435455
    LBA48  user addressable sectors:  769208076
    Logical  Sector size:                   512 bytes
    Physical Sector size:                   512 bytes
    Logical Sector-0 offset:                  0 bytes
    device size with M = 1024*1024:      375589 MBytes
    device size with M = 1000*1000:      393834 MBytes (393 GB)
    cache/buffer size  = unknown
    Form Factor: 2.5 inch
    Nominal Media Rotation Rate: Solid State Device
Capabilities:
    LBA, IORDY(can be disabled)
    Queue depth: 32
    Standby timer values: spec'd by Standard, with device specific minimum
    R/W multiple sector transfer: Max = 16  Current = 16
    Advanced power management level: 254
    DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
         Cycle time: min=120ns recommended=120ns
    PIO: pio0 pio1 pio2 pio3 pio4 
         Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
    Enabled Supported:
       *    SMART feature set
            Security Mode feature set
       *    Power Management feature set
       *    Write cache
       *    Look-ahead
       *    WRITE_BUFFER command
       *    READ_BUFFER command
       *    NOP cmd
       *    DOWNLOAD_MICROCODE
       *    Advanced Power Management feature set
       *    48-bit Address feature set
       *    Mandatory FLUSH_CACHE
       *    FLUSH_CACHE_EXT
       *    SMART error logging
       *    SMART self-test
       *    General Purpose Logging feature set
       *    WRITE_{DMA|MULTIPLE}_FUA_EXT
       *    64-bit World wide name
       *    IDLE_IMMEDIATE with UNLOAD
            Write-Read-Verify feature set
       *    WRITE_UNCORRECTABLE_EXT command
       *    {READ,WRITE}_DMA_EXT_GPL commands
       *    Segmented DOWNLOAD_MICROCODE
            unknown 119[8]
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    Native Command Queueing (NCQ)
       *    Phy event counters
       *    NCQ priority information
       *    READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
       *    DMA Setup Auto-Activate optimization
            Device-initiated interface power management
       *    Software settings preservation
            Device Sleep (DEVSLP)
       *    SMART Command Transport (SCT) feature set
       *    SCT Write Same (AC2)
       *    SCT Features Control (AC4)
       *    SCT Data Tables (AC5)
       *    reserved 69[3]
       *    reserved 69[4]
       *    reserved 69[7]
       *    DOWNLOAD MICROCODE DMA command
       *    WRITE BUFFER DMA command
       *    READ BUFFER DMA command
       *    Data Set Management TRIM supported (limit 8 blocks)
       *    Deterministic read ZEROs after TRIM
Security: 
    Master password revision code = 65534
        supported
    not enabled
    not locked
    not frozen
    not expired: security count
        supported: enhanced erase
    2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT. 
Logical Unit WWN Device Identifier: 500a075113837e16
    NAA     : 5
    IEEE OUI    : 00a075
    Unique ID   : 113837e16
Device Sleep:
    DEVSLP Exit Timeout (DETO): 50 ms (drive)
    Minimum DEVSLP Assertion Time (MDAT): 10 ms (drive)
Checksum: correct

Output of fdisk -l /dev/sdc:

Disk /dev/sdc: 366.8 GiB, 393834534912 bytes, 769208076 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Output of cat /sys/block/sdc/queue/scheduler:

noop [deadline] cfq

Output of dmesg | grep "ahci\|ncq":

[    5.490677] ahci 0000:00:1f.2: version 3.0
[    5.490901] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 1.5 Gbps 0x2 impl SATA mode
[    5.498675] ahci 0000:00:1f.2: flags: 64bit ncq sntf led clo pio slum part ems apst 
[    5.507315] scsi host1: ahci
[    5.507435] scsi host2: ahci
[    5.507529] scsi host3: ahci
[    5.507620] scsi host4: ahci
[    5.507708] scsi host5: ahci
[    5.507792] scsi host6: ahci
[   14.382326] Modules linked in: ioatdma(+) ipmi_si(+) ipmi_msghandler mac_hid shpchp lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi coretemp ip_tables x_tables autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 ses enclosure scsi_transport_sas crct10dif_pclmul crc32_pclmul ghash_clmulni_intel igb aesni_intel hid_generic dca aes_x86_64 lrw ptp glue_helper ablk_helper ahci usbhid cryptd pps_core wmi hid libahci megaraid_sas i2c_algo_bit fjes

Looking deeper at dmesg output, the following strange messagges were bolded and quite suspicious:

...
[    0.081418] CPU: Physical Processor ID: 0
[    0.081421] CPU: Processor Core ID: 0
[    0.081427] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.081430] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.081434] mce: CPU supports 20 MCE banks
[    0.081462] CPU0: Thermal monitoring enabled (TM1)
...
[    0.341838] cpuidle: using governor menu
[    0.341841] PCCT header not found.
[    0.341868] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.341873] ACPI: bus type PCI registered
...
[    1.313494] NET: Registered protocol family 1
[    1.313857] pci 0000:16:00.0: [Firmware Bug]: VPD access disabled
[    1.314223] pci 0000:04:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
...
[    1.591739] PCI: Probing PCI hardware (bus 7f)
[    1.591761] ACPI: \: failed to evaluate _DSM (0x1001)
[    1.591764] PCI host bridge to bus 0000:7f
...
[    1.595018] PCI: root bus ff: using default resources
[    1.595019] PCI: Probing PCI hardware (bus ff)
[    1.595039] ACPI: \: failed to evaluate _DSM (0x1001)
...
[    1.854466] ACPI: Power Button [PWRF]
[    1.855209] ERST: Can not request [mem 0x7e908000-0x7e909bff] for ERST.
[    1.855492] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
...

UPDATE 2

My question isn't a duplicate of this question because my IOPS are always half the expected IOPS for single SSDs and not for the whole RAID, even at low IO Depths where IOPS are very small (<10k).

Look at the graph above: at IO Depth of 1 the single SSDs hit an average of 5794 while they should be at least 8000 each, which is far away from my top edge limit of 40k. I didn't write down RAID results because they are aligned with expected behaviour, but here it is: RAID10 reaches around 120k IOPS for IO Depth of 16 and 32 (about 40k IOPS per 6 disks over 2 due to RAID10 mirroring penalty, so 40k for 3).

I think too that my embedded RAID card can represent the bottle-neck but I can't find a definitive answer. I observe, for example, that running a fio test-run over each SSDs in parallel (6 tests running at the same time, each of them on one single SSD) halves the IOPS of single SSD for IO Depth of 16 and 32. This bring the IOPS at 20k where they were 40k.