LSI RAID: Write cache policy affects read performance

I think that "The LSI card does not have a BBU (yet) so write-back is disabled" is the bottleneck.

If you have UPS - enable the Write-Back.

If not - try to get the BBU.

If you can't - you can either risk the data consistency on the virtual drive by loosing the cached data in case of power surge if you enable Write-Back or stick to these speeds using write through cache.

Even if you align the partition to the logical volume (which is normally automatically done by most modern OSes) and format the volume with optimized cluster/block size big enough (i think it should be 2mb in your case) to get all the drives to process the single IO request i don't think you will achieve very big write performance difference.

Because the write performance of the RAID5 is very over-headed process. And since it is write through the XOR processor don't have the whole data in cache to perform the parity calculations in real time i think

With Write-Back enabled cache on 4x320gb hdds 515kb stip sized RAID 5 i get average 250-350 MB/s write speed writing big sequential files or average 150 MB/s copying big files inside the virtual volume. (i still don't have BBU but i have and old apc 700VA smart ups so i think its enough to minimize the power surges and eventual cache loss to a lot)

Are we discussing 100% random, 100% sequential or some mixed pattern? I am mostly experiencing high speeds when i fully read, write or copy big files on/from/to my array. On the other hand as already said random writes (reads) are much lower variating from less than 1 mb/s up to 190 mb/s average speeds depending on the file sizes and/or request sizes. Mostly under the 20mb/s range in everyday small size/file uses. So it depends a lot from the applications in the real life random transfers. As i am using windows OS my volumes are pretty mush as de-fragmented and for big files big operations like copying from/to are pretty fast

And one suggestion as a solution to the slow read/write random speeds of normal hdds - if you get to the point of reconfiguring your whole controller configuration why don't you consider CacheCade using 1 or 2 of the SSDs for no-power-dependent raid cache (something like the adaptec hybrid raids) and the rest for your OS/app drive as you are using them now? This way you should be able to boost the speed of your raid 5 volume even with write through i think because the actual write to the physical hdds should take place in the background and as you are using write through cache (no on board controller cache) and the ssds as cache instead i think you should be worry free of system resets. But for actual and concrete information on how cachecade works please read lsi's documentation and even ask LSI's technical support as i haven't got the chance to use it yet.

Your limitation here is your raid controller. Random writes like a large cache, and parity-based RAIDs like a fast parity calculator. The more disks you add to a parity-based RAID array (like RAID 5 or 6) on a locally attached controller, the lower performance you'll see per spindle. Large direct attached storage (above 8 drives or so) tends to be raid 10 to avoid this issue. The only reason to use parity based RAID (5 or 6) is to benefit from a higher ratio of usable space. The downside is that if you lose a drive, the rebuild will take more time if the RAID was a large one.

No new hardware

If your hardware is fixed and you need to make this work as well as possible, then your best performance would be from RAID 10, however you'd lose half the available space. Also, while it seems more resilient at first because you can lose up to half the drives without a failure, you can still lose data if you lose the wrong two drives.

The other option for locally attached storage is ZFS. I don't claim to know its inner workings, but my understanding is that it will ignore the RAID card completely and work on the underlying disks. It might have a lower penalty on the parity for small writes if you configure it properly.

New hardware

If you have money to fix this problem, you would be well served to invest in a better RAID controller. Something with more cache and a faster processor. If you decide to use parity based RAID, you would be best served by making multiple RAIDs- preferably 3 groups with 8 spindles each.

Even better than this would be getting some external storage: something that has a storage controller which handles all the caching and parity calculation, and simply allows access to the storage via SAS or fibre channel.