Force RAID 1+0 HP Smart Array to rebuild

hphp-prolianthp-smart-arrayraid10storage

I have a RAID10 with status "Ready for Rebuild", but how can I force the rebuild?

# hpacucli ctrl all show config 

Smart Array E200 in Slot 0 (Embedded)     (sn: PA6C90N9SW82IU)

array A (SAS, Unused Space: 0  MB)

  logicaldrive 1 (68.3 GB, RAID 1, OK)

  physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)
  physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)

array B (SATA, Unused Space: 0  MB)

  logicaldrive 2 (2.0 TB, RAID 1+0, Ready for Rebuild) <---

  physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 750 GB, OK)
  physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 750 GB, OK)
  physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 750 GB, OK)
  physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 750 GB, OK)
  physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 750 GB, OK)
  physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 750 GB, OK)

Details about the RAID1+0:

# hpacucli controller slot=0 logicaldrive 2 show 

Smart Array E200 in Slot 0 (Embedded)

array B

  Logical Drive: 2
     Size: 2.0 TB
     Fault Tolerance: RAID 1+0
     Heads: 255
     Sectors Per Track: 32
     Cylinders: 65535
     Strip Size: 128 KB
     Full Stripe Size: 384 KB
     Status: Ready for Rebuild  <----
     Caching:  Enabled
     Unique Identifier: 600508B100104E395357383249550009
     Disk Name: /dev/cciss/c0d1
     Mount Points: None
     OS Status: LOCKED
     Logical Drive Label: A00E9989PA6C90N9SW82IU44B0
     Mirror Group 0:
        physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 750 GB, OK)
        physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 750 GB, OK)
        physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 750 GB, OK)
     Mirror Group 1:
        physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 750 GB, OK)
        physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 750 GB, OK)
        physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 750 GB, OK)
     Drive Type: Data

This is the controller:

RAID bus controller: Hewlett-Packard Company Smart Array E200i (SAS Controller)

Thanks a lot!

— edit

As requested, follow the 'crtl all show config' using Firmware Version: 1.78:

# hpacucli ctrl all show config detail

Smart Array E200 in Slot 0 (Embedded)
Bus Interface: PCI
Slot: 0
Serial Number: PA6C90N9SW82IU
Cache Serial Number: P9A3A0B9SW8426
RAID 6 (ADG) Status: Disabled
Controller Status: OK
Hardware Revision: A
Firmware Version: 1.78
Rebuild Priority: Medium
Expand Priority: Medium
Surface Scan Delay: 15 secs
Surface Scan Mode: Idle
Post Prompt Timeout: 0 secs
Cache Board Present: True
Cache Status: OK
Cache Ratio: 50% Read / 50% Write
Drive Write Cache: Disabled
Total Cache Size: 128 MB
Total Cache Memory Available: 96 MB
No-Battery Write Cache: Disabled
Cache Backup Power Source: Batteries
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
SATA NCQ Supported: False

Array: A
Interface Type: SAS
Unused Space: 0 MB
Status: OK
Array Type: Data

  Logical Drive: 1
     Size: 68.3 GB
     Fault Tolerance: RAID 1
     Heads: 255
     Sectors Per Track: 32
     Cylinders: 17562
     Strip Size: 128 KB
     Full Stripe Size: 128 KB
     Status: OK
     Caching:  Enabled
     Unique Identifier: 600508B100104E395357383249550006
     Disk Name: /dev/cciss/c0d0
     Mount Points: /boot 99 MB
     OS Status: LOCKED
     Logical Drive Label: A0081154PA6C90N9SW82IUBFAF
     Mirror Group 0:
        physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SAS, 72 GB, OK)
     Mirror Group 1:
        physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SAS, 72 GB, OK)
     Drive Type: Data

  physicaldrive 2I:1:1
     Port: 2I
     Box: 1
     Bay: 1
     Status: OK
     Drive Type: Data Drive
     Interface Type: SAS
     Size: 72 GB
     Rotational Speed: 15000
     Firmware Revision: HPD0
     Serial Number: BJL4P86004VM0826
     Model: HP      DF072BAFDT      
     PHY Count: 2
     PHY Transfer Rate: 3.0Gbps, Unknown

  physicaldrive 2I:1:2
     Port: 2I
     Box: 1
     Bay: 2
     Status: OK
     Drive Type: Data Drive
     Interface Type: SAS
     Size: 72 GB
     Rotational Speed: 15000
     Firmware Revision: HPD0
     Serial Number: BJL4P86004RY0826
     Model: HP      DF072BAFDT      
     PHY Count: 2
     PHY Transfer Rate: 3.0Gbps, Unknown

Array: B
Interface Type: SATA
Unused Space: 0 MB
Status: OK
Array Type: Data

  Logical Drive: 2
     Size: 2.0 TB
     Fault Tolerance: RAID 1+0
     Heads: 255
     Sectors Per Track: 32
     Cylinders: 65535
     Strip Size: 128 KB
     Full Stripe Size: 384 KB
     Status: Ready for Rebuild
     Caching:  Enabled
     Unique Identifier: 600508B100104E395357383249550009
     Disk Name: /dev/cciss/c0d1
     Mount Points: None
     OS Status: LOCKED
     Logical Drive Label: A00E9989PA6C90N9SW82IU44B0
     Mirror Group 0:
        physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 750 GB, OK)
        physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 750 GB, OK)
        physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 750 GB, OK)
     Mirror Group 1:
        physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 750 GB, OK)
        physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 750 GB, OK)
        physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 750 GB, OK)
     Drive Type: Data

  physicaldrive 1I:1:5
     Port: 1I
     Box: 1
     Bay: 5
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPG1    
     Serial Number: 9QK117ZV            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

  physicaldrive 1I:1:6
     Port: 1I
     Box: 1
     Bay: 6
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPG1    
     Serial Number: 9QK10N2K            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

  physicaldrive 1I:1:7
     Port: 1I
     Box: 1
     Bay: 7
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPGB    
     Serial Number: 9QK0X1DQ            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

  physicaldrive 1I:1:8
     Port: 1I
     Box: 1
     Bay: 8
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPGB    
     Serial Number: 9QK2QK4H            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

  physicaldrive 2I:1:3
     Port: 2I
     Box: 1
     Bay: 3
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPG1    
     Serial Number: 9QK119T8            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

  physicaldrive 2I:1:4
     Port: 2I
     Box: 1
     Bay: 4
     Status: OK
     Drive Type: Data Drive
     Interface Type: SATA
     Size: 750 GB
     Firmware Revision: HPG1    
     Serial Number: 9QK10NMD            
     Model: ATA     GB0750C8047     
     SATA NCQ Capable: False
     PHY Count: 1
     PHY Transfer Rate: 1.5Gbps

Best Answer

You can't.

"Ready for Rebuild" status means that the rebuild is likely running into a read error from a partner disk. You may have another failing disk in the array. See: Force LUN in a HP Smart Array to rebuild

Did you recently replace a disk?
Please provide the firmware revision of the Smart Array E200i controller you're using.
Please show ctrl all show config detail

I'd check your backups and schedule time to power the system off and back on again. This could be a firmware issue, depending on what happened before this, or you may have the failing disk situation described above.

Related Solutions

HP SmartArray P400 – How to Repair Failed Logical Drive

The answer is not pleasant. There's a high probability that your array is in a "waiting for rebuild" state, where there's another failing disk in the RAID5 array set that's preventing the recovery from completing. This is why you should avoid RAID5 these days. It doesn't help that these are SATA drives... The likelihood of problems is even higher. Try powering the system off (letting the drives spin down) and powering back on. Follow the prompts at the BIOS array screen and choose the F2 option to "reenable all logical drives". This may kickstart the rebuild process.

Otherwise, it's a rebuild/recovery with new disks.

HP DL380 G7 – Poor RAID 10 Performance with Smart Array P410i

Your system is definitely underperforming based on your hardware specifications. I loaded the sysbench utility on a couple of idle HP ProLiant DL380 G6/G7 servers running CentOS 5/6 to check their performance. These are normal fixed partitions instead of LVM. (I don't typically use LVM, because of the flexibility offered by HP Smart Array controllers)

The DL380 G6 has a 6-disk RAID 1+0 array on a Smart Array P410 controller with 512MB of battery-backed cache. The DL380 G7 has a 2-disk enterprise SLC SSD array. The filesystems are XFS. I used the same sysbench command line as you did:

sysbench --init-rng=on --test=fileio --num-threads=16 --file-num=128 --file-block-size=4K --file-total-size=54G --file-test-mode=rndrd --file-fsync-freq=0 --file-fsync-end=off --max-requests=30000 run

My results were 1595 random reads-per-second across 6-disks.
On SSD, the result was 39047 random reads-per-second. Full results are at the end of this post...

As for your setup, the first thing that jumps out at me is the size of your test partition. You're nearly filling the 60GB partition with 54GB of test files. I'm not sure if ext4 has an issue performing at 90+%, but that's the quickest thing for you to modify and retest. (or use a smaller set of test data)
Even with LVM, there are some tuning options available on this controller/disk setup. Checking the read-ahead and changing the I/O scheduler setting from the default cfq to deadline or noop is helpful. Please see the question and answers at: Linux - real-world hardware RAID controller tuning (scsi and cciss)
What is your RAID controller cache ratio? I typically use a 75%/25% write/read balance. This should be a quick test. The 6-disk array completed in 18 seconds. Yours took over 2 minutes.
Can you run a bonnie++ or iozone test on the partition/array in question? It would be helpful to see if there are any other bottlenecks on the system. I wasn't familiar with sysbench, but I think these other tools will give you a better overview of the system's capabilities.
Filesystem mount options may make a small difference, but I think the problem could be deeper than that...

hpacucli output...

Smart Array P410i in Slot 0 (Embedded)    (sn: 50123456789ABCDE)

   array A (SAS, Unused Space: 0 MB)

      logicaldrive 1 (838.1 GB, RAID 1+0, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 300 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 300 GB, OK)
      physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 300 GB, OK)
      physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 300 GB, OK)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 (WWID: 50123456789ABCED)

sysbench DL380 G6 6-disk results...

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.

Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed:  30001 Read, 0 Write, 0 Other = 30001 Total
Read 117.19Mb  Written 0b  Total transferred 117.19Mb  (6.2292Mb/sec)
 1594.67 Requests/sec executed

Test execution summary:
    total time:                          18.8133s
    total number of events:              30001
    total time taken by event execution: 300.7545
    per-request statistics:
         min:                                  0.00ms
         avg:                                 10.02ms
         max:                                277.41ms
         approx.  95 percentile:              25.58ms

Threads fairness:
    events (avg/stddev):           1875.0625/41.46
    execution time (avg/stddev):   18.7972/0.01

sysbench DL380 G7 SSD results...

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 432Mb each
54Gb total file size
Block size 4Kb
Number of random requests for random IO: 30000
Read/Write ratio for combined random IO test: 1.50
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed:  30038 Read, 0 Write, 0 Other = 30038 Total
Read 117.34Mb  Written 0b  Total transferred 117.34Mb  (152.53Mb/sec)
39046.89 Requests/sec executed

Test execution summary:
    total time:                          0.7693s
    total number of events:              30038
    total time taken by event execution: 12.2631
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.41ms
         max:                                  1.89ms
         approx.  95 percentile:               0.57ms

Threads fairness:
    events (avg/stddev):           1877.3750/15.59
    execution time (avg/stddev):   0.7664/0.00

Best Answer

Related Solutions

HP SmartArray P400 – How to Repair Failed Logical Drive

HP DL380 G7 – Poor RAID 10 Performance with Smart Array P410i

Related Topic