NFS on top of GFS2 – does it work

gfs2nfs

We're currently using a NoSQL derivative called Splunk to receive our data. The software supports something called "search head pooling" in which the job-dispatching engine is housed on several servers which share a common storage point. Originally our intention was to use a clustered filesystem like GFS2 because of low latency, stability, and ease of setup. We set up GFS2, and it's working with no issues.

However when trying to run the software, it's trying to create lock files, and a bunch of other things that their support team can't quite explain. Ultimate feedback from them was that they only support NFS.

Our network administration team heavily frowns on NFS (lack of stability, file lock issues, etc).

So, I was thinking about the possibility of setting up NFS on each server in the cluster to act as a wedge layer between the GFS2 filesystem and the software. Basically configure each server to export the GFS2 filesystem's mountpoint via NFS, and then tell each server to connect to that NFS share. That way we aren't introducing any single-points-of-failure should a dedicated NFS server go down, but the vendor gets their "required" NFS share.

I'm just brainstorming ways around, so please tear this apart 🙂

Best Answer

The way GFS2 locking works, you may be seeing serious performance problems by pointing each node to a different NFS server:

If another node requests a glock which cannot be granted immediately, then the DLM sends a message to the node or nodes which currently hold the glocks blocking the new request to ask them to drop their locks. Dropping glocks can be (by the standards of most file system operations) a long process. Dropping a shared glock requires only that the cache be invalidated, which is relatively quick and proportional to the amount of cached data.

Dropping an exclusive glock requires a log flush, and writing back any changed data to disk, followed by the invalidation as per the shared glock.

[...]

Due to the way in which GFS2's caching is implemented the best performance is obtained when either of the following takes place:

  • An inode is used in a read only fashion across all nodes.
  • An inode is written or modified from a single node only.

Additionally, support documents as those from Red Hat indicate that POSIX locks on NFS shares would induce problems, so only Active/Passive clustering configurations where NFS is exported from a single active node at any given time and no file access to the GFS2 file system is performed except the one set up through the NFS service would be supported. Obviously, this should take care of any kind of unforeseen locking interactions between NFS and GFS2, but probably is not what you wanted to see.

Related Topic