Best Performance for Max Files in a Directory on Linux File System

apache-2.2filesystemslinuxPHP

how many files in a diretory will be decrease server performance? I have a website that contain hundreds thousand of images ( >a million in separated directory). I wonder if it will effect the performance.

server detail: centos, apache, php 5

Best Answer

There isn't an easy way to answer the question, but take a look at things like:

  • /usr/share/lib/terminfo/...
  • CPAN authors/id/...

In both cases, with far fewer than a million entries, the designers split the directories into multiple levels to speed up access.

If you have a million entries and the file system does not have any search structure built into the directory handling code, then accessing a file is going to require the o/s to read about half the name + inode number entries in the directory for each file. Even if it is all in the buffer pool, that becomes a significant workload.

If you introduce a tiered naming system - both the examples base it on the first characters of the name:

 terminfo/lib/a/ansi
 id/J/JO/JOHNL

CPAN has two levels; for your 1 million files, I'd probably use two levels too.

There is some overhead in having the extra level(s) of directory.

These schemes assume you know the one name you are looking for - searching through all names is a different proposition.