I've got an EXT3 formatted drive on a Linux CentOS server. This is a web app data drive and contains a directory for every user account ( there are 25,000 users ). Each folder contains files that that user has uploaded. Overall, this drive has roughly 250GB of data on it.
Does structuring the drive with all these directories impact drive read/write performance? Does it impact some other performance aspect I'm not aware of?
Is there anything inherently wrong or bad with structuring things this way? Perhaps just the wrong choice of filesystem?
I've recently tried merging two data drives and realized that EXT3 is limited to 32,000 subdirectories. This got me wondering why. It seems silly that I built it this way, considering each file has a unique id that corresponds to an id in the database. Alas …
Best Answer
This is easy to test the options for yourself, in your environment and compare the results. Yes, there is a negative impact on performance as the number of directories increases. Yes, other filesystems can help get around those barriers or reduce the impact.
The XFS filesystem is better for this type of directory structure. ext4 is probably just fine nowadays. Access and operations on the directory will simply slow down as the number of subdirectories and files increase. This is very pronounced under ext3 and not so much on XFS.