Nfs – Performance stats for ESXi VMs on Openfiler NFS

There are many reasons why transferring many small files will always be slower than transferring a single large file. For a read, the files are more likely to be scattered around the disk, requiring seeks all over the place to get them. As Evan mentioned, there's also metadata involved in the case of NFS (or any other file system for that matter!) which also complicates things.

You can try increasing your rsize and wsize parameters to the NFS mount and see if that will help performance a bit. Also check out this question on tuning NFS for minimum latency as it has a lot of helpful advice that will help in the case of many small file transfers.

I would think hard about using sync. I'm reminded about the Dirty Harry 'Do you feel lucky' speech.

NFS v2 and v3 is designed such that when the write is acked from the server to the client the data is on disk. This allows NFS to be stateless, and, therefore, the server COULD reboot between each request. One hopes not, but it could happen.

This means that if the client sees the ACK on the write then it doesn't have to care anymore about the data being on disk.

If you use async then this is not true anymore. It will be way faster though. Basically though if you use async, and, the server crashes, you probably should reboot the clients unless you know exactly what they do since if they are expecting to keep multiple files in sync the client may believe they are in sync, when, in fact, they are not.