Linux – nfs file access statistics

analysislinuxnfs

For a Linux NFSv3 server, is there any way to see statistics on actual file access? In other words, I would like to see a timestamped log of every file read or written.

Motivation: I have several NFS servers that are slated to be replaced by some kind of "big iron" storage system. In my particular case, the NFS client load is virtually all reads of large files (averaging around 700 MB). One vendor's solution uses a large RAM disk as a cache to provide the kind of massive random read throughput that I need.

What I want is a way to study what the NFS file access pattern looks like so that I can "right size" the cache.

Best Answer

I've been banging my head against this problem all day, with no solution found.

  1. You can turn up the debugging of the NFS server, but that doesn't provide much detail (if that exmaple is accurate) and will probably dominate a busy NFS server's disk with useless baggage logged in addition to the bare filenames.

  2. Another solution is adding rules to auditd/auditctl to log all reads or writes to the NFS directories, but that doesn't work for our Centos 6.X machines, for reasons I can't quite figure out yet. In /etc/audit/audit.rules on a client machine:

    # First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 8192

# Feel free to add below this line. See auditctl man page
-w /auto/ -p r -k read -k home
-w /auto/ -p w -k write -k home
-w /auto/ -p xa -k other -k home

...where I've given separate keys to reading, writing, and executing/changing attributes. My clients are autofs'd to mount a few different NFS directories, including their home directory, to /auto/ with soft links pointing the client machine's /home/users/ back to /auto/. I get logging of lots of stuff, but none of the files the users themselves seem to be modifying.

Troll the audit logs with ausearch -k read | aureport -f, for instance. grepping for .ODT or .PDF comes up with nothing, the only results are for metacity's configs, Chrome's crap, etc., etc.

Naturally, enabling audit on the server pointing at the real /home/users/XYZ only shows accesses from things interfacing with the server directly (mail clients) or users SSH'd into the server.

If you can figure out the right recipe for audit, or a dedicated solution all together, please, please, please share it! You'd think this would have been solved in 1993.

Related Topic