Lvm – mdadm+zfs vs mdadm+lvm

lvmmdadmsoftware-raidzfs

This may be a naive question since I'm new to this and I cannot find any results about mdadm+zfs, but after some testing it seems it might work:

The use case is a server with RAID6 for some data that is backed-up somewhat infrequently. I think I'm well served by any of ZFS or RAID6. Platform is Linux. Performance is secondary. So the two setups I am considering are:

  • A RAID6 array plus regular LVM and ext4
  • A RAID6 array plus ZFS (without redundancy). Is this second option that I don't see discussed at all.

Why ZFS+RAID6? It's mainly because the inability of ZFS to grow a raidz2 with new disks. You can replace disks with larger ones, I know, but not add another disk. You can accomplish 2-disk redundancy and ZFS disk growth using mdadm as the redundancy layer.

Besides that main point (otherwise I could go directly to raidz2 without RAID under it), these are the pros-cons that I see for each option:

  • ZFS has snapshots without preallocated space. LVM requires preallocation (might be no longer true).
  • ZFS has checksumming (very interested in this) and compression (nice bonus).
  • LVM has online filesystem growth (ZFS can do it offline with export/mdadm –grow/import).
  • LVM has encryption (ZFS-on-Linux has not). This is the only major con of this combo I see. I guess I could go RAID6+LVM+ZFS… seems too heavy, or not?

So, to close with a proper question:

1) Is there anything that inherently discourages or precludes RAID6+ZFS? Anyone has experience with a setup like this?

2) Are there possibilities for checksumming and compression that would make ZFS unnecessary (maintaining the possibility of filesystem growth)? Because the RAID6+LVM combo seems the sanctioned, tested way.

Best Answer

If you go for the MDADM raid6 + ZFS solution, you will lose self healing as ZFS have no data about how to recalculate/restore damaged data blocks, unless if you set copies=2 or more(which will use twice as much storage). Your ZFS storage will only report data inconsistencies.

That means if a data block cannot be properly read, ZFS will tell you which files are affected. Your whole file system shouldn't be in danger as meta data is dual redundant and can be self healed if any inconsistencies are detected.

You can also grow your ZFS file system by adding more vdevs to your pool, which will be dynamically striped. For example you can create a new raidz2 vdev with 4 to 8 disks and add it to your pool. You can even add a mirrored vdev to it.

Related Topic