I have a weird issue with one of the NFS clients (Ubuntu 16.04 LTS).
I have been working for the past few days to try and debug this, but so far without success.
After I mount the partition, everything works perfectly for a few days, transfer speed is 1 Gbps between the client and server. After a few days the speed drops to less than 10 mbps and even a simple directory listing takes a few seconds and I/O wait is at 100%
What I noticed is that the backlog wait, especially for the write operations is extremely high:
root@srv:~# mountstats /mnt/data
Stats for 192.168.0.15:/mnt/data mounted on /mnt/data:
NFS mount options: rw,vers=4.0,rsize=16384,wsize=16384,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.150,local_lock=none
NFS server capabilities: caps=0xffdf,wtmult=512,dtsize=16384,bsize=0,namlen=255
NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,bm2=0x0,acl=0x3,pnfs=notconfigured
NFS security flavor: 1 pseudoflavor: 0
NFS byte counts:
applications read 8168679407142 bytes via read(2)
applications wrote 4833000353435 bytes via write(2)
applications read 0 bytes via O_DIRECT read(2)
applications wrote 0 bytes via O_DIRECT write(2)
client read 4218977852758 bytes via NFS READ
client wrote 4832098253207 bytes via NFS WRITE
RPC statistics:
561421762 RPC requests sent, 561421608 RPC replies received (1 XIDs not found)
average backlog queue length: 0
READ:
263822474 ops (46%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 184 avg bytes received per op: 16051
backlog wait: 8.772689 RTT: 27.972131 total execute time: 36.752241 (milliseconds)
WRITE:
295296111 ops (52%) 0 retrans (0%) 0 major timeouts
avg bytes sent per op: 16567 avg bytes received per op: 132
backlog wait: 62468603019.791718 RTT: 78.030143 total execute time: 62468603097.830574 (milliseconds)
There are no errors, no warnings, I tried debugging with "echo 1 > /proc/sys/vm/block_dump" (which has done miracles for me in the past), but this time there's nothing visible related to NFS.
Any idea how I can debug further and see what's causing the extremely high backlog wait?
Best Answer
Just in case anyone has the same problem, the only fix I could find was to force NFS3 instead NFS4. The problem has disappeared now.