Should I use ZFS inside a virtual machine? According to existing information (in particular this thread, which I will quote below) it is not a good idea, unless certain measures are provided (such as using VT-d). The basic problem allegedly is that the hypervisor may report a block to be written when in fact, it has not yet been written. Allegedly, ZFS is particularly sensitive to such a discrepancy, compared to other filesystems:
The specifics of why it's so bad relate to it's fsck-less design.
It was also acknowledged in that discussion that:
The only scenario where "pointer written before data" turns into "pointer written but no data" is still the event of a crash of either OS or the hypervisor, as far as I can imagine.
But what is the worst-case consequence in case of such a crash?
- Corruption or loss of files written around the time of the crash?
- Corruption or loss of any other file in the dataset?
- Corruption or loss of any file in a dataset that was not being modified around the time of the crash, e.g., a snapshot that was taken long before the crash?
Is there any silent corruption possible, which cannot be detected by a scrub?
I have tried to simulate a crash by using VirtualBox and selecting Close / Power Off while data was written to a ZFS inside of a VM. I did this about 20 times, but could not detect any problems. Maybe this method of testing is not appropriate. Are there better ways to experiment with this?
Best Answer
Basically any modern hypervisor (VMWare, Xen, KVM, HyperV, even VirtualBox) supports barrier passing: when a VM explicitly flushes something to disk (by issuing a barrier/FUA), the hypervisor will pass the barrier down to the host, forcing the same flushes executed by the guest OS. In other words, no corruption is expected for important/durable writes (as the one used by the filesystem itself to update its metadata).
While most hypervisor can be confiured to ignore flushes, this can jeopardize any filesystems - XFS, EXT4, etc. will be exposed to serious corruptions as much as ZFS.
Back to the main question: it is perfectly safe to use ZFS inside a guest OS, and I have first-hand experience on similar setup. This will enable the guest to use advanced features as compression, snapshots and send/receive. However this can led to somewhat lower guest performance (ZFS is not enginereed to be a benchmark-winning filesystem). Moreover, as many SANs implements the very same features at the disk image level, you should evaluate if the performance impact due to double-CoW is worth the additional flexibility at the guest level.
For the reasons above, I generally use ZFS at the disk-image/hypervisor level, while using XFS or EXT4 inside the virtual machines themselves. However, on scenario where I have no visibility on the underlying SAN/storage (and its snapshot/compression/replication policies), I sometime use ZFS at the guest level.
In these cases, the added features more than compensate the performance impact vs, for example, a plain XFS setup. And I have no stability/durability problems at all.
Side note: VT-d is only useful if you plan to pass the raw disks (or other hardware devices) to the guest itself. If using file or volume based virtual disk, you are not using VT-d.