Linux – LVM-on-RAID vs RAID10 vs RAID6

linuxlvmmdadmsoftware-raid

In the spirit of the question LVM mirroring VS RAID1 I thought I'd push a bit further.

Suppose I have 4 identical hard drives (e.g. 1TB) that I want in a RAID setup for a backup/media server. LVM over the top seems to be current best practice for resizing, but what about the RAID layout? My options seem to be:

  • 2x 1TB RAID1 sets with LVM concatenation
  • 2x 1TB RAID1 sets with LVM striping
  • 1x 2TB RAID10 (striped 2x 1TB) set with LVM on top
  • 1x 2TB RAID6 (striped 2x1TB with 2x 1TB parity) with LVM on top

RAID-on-LVM sounds too crazy to be worth investigating, but I'd be interested if someone can make a good case for using it.

How does each of these perform in the scenario of

  • The I/O load will be a combination of low but constant (e.g. video streaming) and high but bursty (e.g. a backup system)
  • It is very likely that a drive will die unexpectedly
  • I want to upgrade (increase the capacity of) the drives 1 or 2 at a time (no longer relevant)
  • At some point in the future I might want to add another 2x HDDs in RAID1 to the LVM

What are you recommendations/experiences with any of these RAID configs?

Best Answer

I would not mix md (software RAID) and md (LVM) RAID features. In the spirit of KISS, I'd go with pure md with LVM on top for snapshots / resizes.

With 4 disks going RAID6 is a Bad Idea (TM). It gives you exactly as much space as RAID 10, but with much, much worse performance (you have to calculate two parities and face read-modify-write penalty for writes smaller than stripe size).

RAID 6 gives you marginally beter resilience (any 2 disks can fail, while in RAID 10 one disk from mirror-pair can fail) at a high cost. Not worth it.

RAID 10 gives you best performance possible in this setup.

  • 2x 1TB RAID1 sets with LVM concatenation -- concurrent I/O performance depends on whether you hit the same, or different disk pairs. Solved by RAID 10, which spreads your I/O over all disks.

  • 2x 1TB RAID1 sets with LVM striping -- should give similar performance to md RAID 10, but you have more complicated setup. I love simplicity.

  • 1x 2TB RAID6 (striped 2x1TB with 2x 1TB parity) with LVM on top -- worse performance for writes, bad performance with a lost drive.

Performance characteristics I'm going to assume RAID 10, because in my opinion it has all advantages and no downsides in your scenario. You are going to be limited by a performance equivalent of a pair of HDDs. In other words you are going to be able to serve/write data at about 2x what a single HDD can do. For streaming you should be able to saturate 1Gbps link with no effort (reading or writing). For bursty data you are stuck with ~150 IOps (assuming 7.2krpm SATA drives). RAID 10 will guarantee that the load is spread among all drives for all I/O (unless you are unlucky enough to have application access data with a stride matching your RAID chunk size), and RAID 10 "far" layout should give you similar performance no matter which region of filesystem you are accessing.

Lost drive means marginal loss in read access time (you loose the "far" layout benefit for the affected mirror pair).

If you expand the storage with another pair of mirrored drives md will not be able to reshape the data to re-stripe it over the new space. Effectively you have a RAID10 + RAID1 setup, unless you backup, re-create the array and restore.

Related Topic