I had an original software RAID 5 setup under Ubuntu 10.04.03, 3 devices, 64k chunk. Performance was acceptable:
# 4GB write
dd if=/dev/zero of=ddfile.big bs=1MB count=4k
4095290816 bytes (4.1 GB) copied, 51.743 s, 79.1 MB/s
# 4GB read
dd if=ddfile.big of=/dev/null
4095290816 bytes (4.1 GB) copied, 62.5932 s, 65.4 MB/s
Then I went to a software RAID 6 with 3 devices (2 were marked as missing, need to copy the data off them), 128k chunk. I have also set up LVM on the new array. Performance was shockingly poor:
# 4GB write
dd if=/dev/zero of=ddfile.big bs=1MB count=4k
4096000000 bytes (4.1 GB) copied, 106.406 s, 38.5 MB/s
# 4GB read
dd if=ddfile.big of=/dev/null
4096000000 bytes (4.1 GB) copied, 129.317 s, 31.7 MB/s
I'm trying to understand what slowed those reads and writes by half. Could it be the LVM? Or the degraded state of the new raid array? Or could it be the chunk size difference (64k vs 128k)? Other than that, I'm using ext4 for the filesystem, no options.
Best Answer
The number of things wrong with this situation is what is shocking here.
A RAID-6 array has two parities. That means for every bit written there are (essentially) two computations made, one for each parity. Using RAID-6 on an array with less than 5 isn't recommended (and on less than 4 is counterproductive!). Essentially you're doing way too much work for no gain whatsoever.
And this is where it REALLY starts to go downhill. You're doing RAID-6 with two devices missing. So you're calculating parity and putting it..nowhere? (Hey anyone with software RAID-6 experience, is this true? Does it still compute? If so, does it get dropped, or does it try to write parity to the drive?)
And any gain you would have gotten from multiple spindles is utterly eliminated because you don't have any other spindles in use.
It's not really so shocking as much as self obvious at this point. You're doing tons of stuff for absolutely no benefit.
So what should you do?
Ask yourself some questions. Why do you want RAID-6? Is it because it's more fault tolerant than RAID-5? If so, you should understand which failure modes RAID-6 protects you against:
It has two parities. This comes into play whenever you suffer a drive failure, then replace the drive, then have another drive fail during rebuild. This commonly happens when you get a bad batch of drives, or when you've had drives continually exposed to high heat (probably higher than you think). In addition, there's the possibility of a Unrecoverable Read Error (URE) about once every 10^14 bits read (that's a bit over 11TB of data). If it happens during a RAID rebuild, well, that's not good, and RAID-6 can protect that.
Tell us more about the array you want to build. Why are you building it, what does the capacity need to be, and what kind of performance do you need from it? With that information, we can give you recommendations.