Nfs – Making NFS behave like a block device

block-devicecachefilesystemsnfs

We're currently working on a virtualization setup, and because of features like being able to grow volumes on our filers rather than having to go through the process resizing block devices etc. we'd like to use NFS volumes for everything, essentially, rather than exposing "block devices" through iSCSI or similar technologies.

This does however pose a little problem, as file system caching suffers quite a lot, as we cannot find a way to instruct the Linux NFS driver to treat an NFS volume as a block device in terms of caching. I know this is sort of counter intuitive as to what NFS was designed for, but I'm wondering if there's any known way, without using for example CacheFS, to make the NFS client believe that it has exclusive accesss to the volume and thus can freely cache as much in virtual memory as it wants? We can technically guarantee this exclusivity, so it's just a question of making the client believe us, essentially 😉

For the record all hosts are running Linux using kernel version 3.2 on a Xen hypervisor likewise running on a version 3.2 kernel.

Best Answer

Virtualization on NFS is a well established technology. There are benefits and disadvantages to NFS and block, but you can't pick and choose them. Deduplication, volume resizing, and thin provisioning on block devices are all mature technologies that work fine. Same for NFS.

Related Solutions

Nfs – Xen and NFS as block device

I don't suppose this for kernel modules or portage trees, is it? That's what I've seen this mechanism used for...

So, sure enough it's easy to have all of your guests have a filesystem image file attached to them as a read-only block device. It's also very straightforward to have that mounted somewhere in the guest (/etc/fstab and all that Jazz). Ownerships you'll presumably take care of in the block device anyway (assuming you're using a filesystem type that stores that metadata -- but if you're using, say, VFAT, ownership is only a mount option away anyway).

The trick is handling updates. Once you've got your "block device" mounted in any guest, nothing can be allowed to update it. It just won't work, because nobody knows that someone else is updating the contents, so everything falls apart. Instead, you need to create a copy of the file with the filesystem image, make whatever changes are needed, and then trigger some sort of update action to make the guests unmount the old "filesystem", then the dom0 can detach the old file and attach the new one, before the guest remounts the filesystem.

In the cases I've used this, we actually had some code in the domU config files (since they're just Python anyway) to find the newest of these block devices and attach that, then the usual boot-time mounts did the right thing. So, for us, the "update process" was "reboot the guest". Whether that works for you, though, is a question I can't answer because I don't know what you're trying to use this for.

Alternately, just have a second NFS server that is only used for supplying these files to your domUs. It's probably easier than all this block device frufru (we had some pretty specific requirements that made it the least-worst option, but I don't expect they apply in your case -- in fact, I know they don't apply in your case, because you've already got an NFS server).

Linux – Force block device (NBD) to bypass the buffer cache

You can achieve this by mounting with the sync-option.

From the mount(8)-man page:

   sync   All I/O to the filesystem should be done synchronously. In  case
          of  media  with  limited number of write cycles (e.g. some flash
          drives) "sync" may cause life-cycle shortening.

Related Topic

Why is there both character device and block device for nvme