Nfs – Unreliable NFS with large number files in a directory

There are two limits that you may have hit.

• inodes (unlikely)

  You've consumed all of the available inodes for that volume. You can confirm this with the commands df -i and maxfiles.

• maxdirsize (more likely)

  ONTAP imposes a limit on size of directories. Size being a product of metadata/hardlinks, not file content. This limit defaults (assuming >6.5) to 1% of your system RAM. It does so to ensure that large directories don't impact system performance. Because linear directory scans require directory data structures to be loaded into memory. Quick overview:

  • You can check a directory size with ls -lkd.
  • Raising maxdirsize for a volume is a bit of a one way operation.
  • Only raise it in small increments.
  • You can't reduce a directory's size by deleting it's contents.
  • I have these two bookmarked links which contain a lot more information.
    • NOW: What is maxdirsize?
    • Communities: maxdirsize Recommended Setting and Performance Implications

If it's neither of these issues then I'd recommend checking and posting your ONTAP logs.

(answering my own question in case anyone finds it while searching for similar.) There are possibly as many as 9 million files in the directory.

Unfortunately can't log in to the server directly, it's an appliance. The only access to the filesystems is via export.

rm -rf didn't seem to work. watching with strace it was hanging.

find woudn't complete, died with no error.

ls -1 never seemed to complete. (I realize now that it attempts to sort the results, ls -1f might have worked eventually).

what did work was a simple perl snippet. I assume c code do the same would work.

 opendir( my $dh,  '/home/foo' ) or die $!
    while ( my $file = readdir $dh ) {
        print "$file\n";
    }