Large disk queue length

hard drivehphp-prolianthp-smart-arrayperformance

A few days ago disk on my server started to have large queue length:

enter image description here

I've changed controller battary recently, HP configuration utility was saying the battery is bad, but after changing the battery nothing changed. HP configuration utility is now saying everything is fine but the queue is still the same

What can I do to eliminate the problem? Maybe I should change the controller?

UPDATE 1 (gtapscott's questions):
1) This is a read queue, I added a separate read queue counter and it matched with overall queue length. Write queue is empty.
2) Avg. disk queue varies from 0 to several hundreds, average value is about 100-200. I'm not sure but I feel this counter acts like there is no controller cache at all.
3) There is 10 disks in RAID-5

UPDATE 2 (ewwhite's post):
Yes, I rebooted the server after the battary change

I have slightly difference interface, here it is:
enter image description here
enter image description here
enter image description here
enter image description here

So cache is enabled on the RAID massive

UPDATE 3:
The problem was in one of the RAID disks as ewwhite suggested

Best Answer

You mentioned that you had the HP Array Configuration utility installed. Hopefully, you have the remainder of the HP System Management agents installed. Did you reboot following the array battery change? If not, that may help. You can also look at the Array Configuration Utility to check the array status. You should see something like the following, showing Accelerator: Enabled:

enter image description here

or...

enter image description here