HP Smart Array P400 Write Cache not Enabling on VMware ESXi 5.5 Running on ProLiant DL380 G6

hphp-prolianthp-smart-arrayraidvmware-esxi

I am trying to get write caching enabled on a Smart Array P400 even though there is no battery backup. We have a large building wide UPS backed up by a generator so the likelihood of loosing power is minimal and the write performance on this VM host server is terrible. I have installed the HP VMware drivers and software and I have used hpssacli to turn on write caching and no-battery write cache:

/opt/hp/hpssacli/bin # ./hpssacli controller slot=1 show config detail

Smart Array P400 in Slot 1
   Bus Interface: PCI
   Slot: 1
   Serial Number: PAFGK0P9VX029O
   Cache Serial Number: PA82C0J9VX12T7
   RAID 6 (ADG) Status: Disabled
   Controller Status: OK
   Hardware Revision: E
   Firmware Version: 7.22
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 100% Read / 0% Write
   Drive Write Cache: Enabled
   Total Cache Size: 256 MB
   Total Cache Memory Available: 208 MB
   No-Battery Write Cache: Enabled
   Battery/Capacitor Count: 0
   SATA NCQ Supported: True
   Number of Ports: 2 Internal only
   Encryption Supported: False
   Driver Version: 3.6.14
   Driver Supports HP SSD Smart Path: False


   Internal Drive Cage at Port 1I, Box 1, OK
      Power Supply Status: Not Redundant
      Serial Number:
      Drive Bays: 4
      Port: 1I
      Box: 1
      Location: Internal

   Physical Drives
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 250 GB, OK, spare)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 250 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 250 GB, OK)


   Internal Drive Cage at Port 2I, Box 1, OK
      Power Supply Status: Not Redundant
      Serial Number:
      Drive Bays: 4
      Port: 2I
      Box: 1
      Location: Internal

   Physical Drives
      physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 250 GB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 250 GB, OK)
      physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 250 GB, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 120 GB, OK)

   Array: A
      Interface Type: SATA
      Unused Space: 0  MB
      Status: OK
      Array Type: Data



      Logical Drive: 1
         Size: 111.8 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 28722
         Strip Size: 128 KB
         Full Stripe Size: 128 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B10010503956583032394F0009
         Logical Drive Label: A0199599PAFGK0P9VX029O81A9
         Drive Type: Data
         LD Acceleration Method: Controller Cache

      physicaldrive 2I:1:1
         Port: 2I
         Box: 1
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 120 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K647T8B25P5U
         Model: ATA     GJ0120CAGSP
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 33
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps


   Array: B
      Interface Type: SATA
      Unused Space: 0  MB
      Status: OK
      Array Type: Data
      Spare Type: dedicated



      Logical Drive: 2
         Size: 931.4 GB
         Fault Tolerance: 5
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 64 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Parity Initialization Status: Initialization Completed
         Unique Identifier: 600508B10010503956583032394F000A
         Logical Drive Label: A01986FDPAFGK0P9VX029O8FA7
         Drive Type: Data
         LD Acceleration Method: Controller Cache

      physicaldrive 1I:1:5
         Port: 1I
         Box: 1
         Bay: 5
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648TAC28P4N
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 35
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

      physicaldrive 1I:1:6
         Port: 1I
         Box: 1
         Bay: 6
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648T8C25MF2
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 34
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

      physicaldrive 2I:1:2
         Port: 2I
         Box: 1
         Bay: 2
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648T8C25MFW
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 35
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

      physicaldrive 2I:1:3
         Port: 2I
         Box: 1
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648T8B25M9W
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 35
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

      physicaldrive 2I:1:4
         Port: 2I
         Box: 1
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648T8C25ML9
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 35
         Maximum Temperature (C): 58
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

      physicaldrive 1I:1:7
         Port: 1I
         Box: 1
         Bay: 7
         Status: OK
         Drive Type: Spare Drive
         Interface Type: SATA
         Size: 250 GB
         Native Block Size: 512
         Firmware Revision: HPG2
         Serial Number: K648T8C25MK0
         Model: ATA     GJ0250EAGSQ
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 31
         Maximum Temperature (C): 57
         PHY Count: 1
         PHY Transfer Rate: 1.5Gbps

But as you can see, the cache ratio is 0% for writes. What am I missing?

Best Answer

It looks like this looks like what you are looking for:

ctrl slot=0 cacheratio=50/50

source: https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/Smart-Array-Controller-P400-cache-ratio/td-p/4018686

Related Solutions

HP SmartArray P400 – How to Repair Failed Logical Drive

The answer is not pleasant. There's a high probability that your array is in a "waiting for rebuild" state, where there's another failing disk in the RAID5 array set that's preventing the recovery from completing. This is why you should avoid RAID5 these days. It doesn't help that these are SATA drives... The likelihood of problems is even higher. Try powering the system off (letting the drives spin down) and powering back on. Follow the prompts at the BIOS array screen and choose the F2 option to "reenable all logical drives". This may kickstart the rebuild process.

Otherwise, it's a rebuild/recovery with new disks.

HP DL380 G7 – Poor RAID 10 Performance with Smart Array P410i

Your system is definitely underperforming based on your hardware specifications. I loaded the sysbench utility on a couple of idle HP ProLiant DL380 G6/G7 servers running CentOS 5/6 to check their performance. These are normal fixed partitions instead of LVM. (I don't typically use LVM, because of the flexibility offered by HP Smart Array controllers)

The DL380 G6 has a 6-disk RAID 1+0 array on a Smart Array P410 controller with 512MB of battery-backed cache. The DL380 G7 has a 2-disk enterprise SLC SSD array. The filesystems are XFS. I used the same sysbench command line as you did:

sysbench --init-rng=on --test=fileio --num-threads=16 --file-num=128 --file-block-size=4K --file-total-size=54G --file-test-mode=rndrd --file-fsync-freq=0 --file-fsync-end=off --max-requests=30000 run

My results were 1595 random reads-per-second across 6-disks.
On SSD, the result was 39047 random reads-per-second. Full results are at the end of this post...

As for your setup, the first thing that jumps out at me is the size of your test partition. You're nearly filling the 60GB partition with 54GB of test files. I'm not sure if ext4 has an issue performing at 90+%, but that's the quickest thing for you to modify and retest. (or use a smaller set of test data)
Even with LVM, there are some tuning options available on this controller/disk setup. Checking the read-ahead and changing the I/O scheduler setting from the default cfq to deadline or noop is helpful. Please see the question and answers at: Linux - real-world hardware RAID controller tuning (scsi and cciss)
What is your RAID controller cache ratio? I typically use a 75%/25% write/read balance. This should be a quick test. The 6-disk array completed in 18 seconds. Yours took over 2 minutes.
Can you run a bonnie++ or iozone test on the partition/array in question? It would be helpful to see if there are any other bottlenecks on the system. I wasn't familiar with sysbench, but I think these other tools will give you a better overview of the system's capabilities.
Filesystem mount options may make a small difference, but I think the problem could be deeper than that...

hpacucli output...

Smart Array P410i in Slot 0 (Embedded)    (sn: 50123456789ABCDE)

   array A (SAS, Unused Space: 0 MB)

      logicaldrive 1 (838.1 GB, RAID 1+0, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
      physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 300 GB, OK)
      physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 300 GB, OK)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 (WWID: 50123456789ABCED)

sysbench DL380 G6 6-disk results...

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.

Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed:  30001 Read, 0 Write, 0 Other = 30001 Total
Read 117.19Mb  Written 0b  Total transferred 117.19Mb  (6.2292Mb/sec)
 1594.67 Requests/sec executed

Test execution summary:
    total time:                          18.8133s
    total number of events:              30001
    total time taken by event execution: 300.7545
    per-request statistics:
         min:                                  0.00ms
         avg:                                 10.02ms
         max:                                277.41ms
         approx.  95 percentile:              25.58ms

Threads fairness:
    events (avg/stddev):           1875.0625/41.46
    execution time (avg/stddev):   18.7972/0.01

sysbench DL380 G7 SSD results...

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed:  30038 Read, 0 Write, 0 Other = 30038 Total
Read 117.34Mb  Written 0b  Total transferred 117.34Mb  (152.53Mb/sec)
39046.89 Requests/sec executed

Test execution summary:
    total time:                          0.7693s
    total number of events:              30038
    total time taken by event execution: 12.2631
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.41ms
         max:                                  1.89ms
         approx.  95 percentile:               0.57ms

Threads fairness:
    events (avg/stddev):           1877.3750/15.59
    execution time (avg/stddev):   0.7664/0.00

Best Answer

Related Solutions

HP SmartArray P400 – How to Repair Failed Logical Drive

HP DL380 G7 – Poor RAID 10 Performance with Smart Array P410i

Related Topic