Don't expect this to run quickly...
cd to a directory where you suspect there might be a subdirectory with lots of inodes. If this script takes a huge amount of time, you've likely found where in the filesystem to look. /var is a good start...
Otherwise, if you change to the top directory in that filesystem and run this and wait for it to finish, you'll find the directory with all the inodes.
find . -type d |
while
read line
do
echo "$( find "$line" -maxdepth 1 | wc -l) $line"
done |
sort -rn | less
I'm not worried about the cost of sorting. I ran a test and sorting through the unsorted output of that against 350,000 directories took 8 seconds. The initial find took . The real cost is opening all these directories in the while loop. (the loop itself takes 22 seconds). (The test data was run on a subdirectory with 350,000 directories, one of which had a million files, the rest had between 1 and 15 directories).
Various people had pointed out that ls is not great at that because it sorts the output. I had tried echo, but that is also not great. Someone else had pointed out that stat gives this info (number of directory entries) but that it isn't portable. It turns out that find -maxdepth is really fast at opening directories and counts .files, so... here it is.. points for everyone!
Check is your version of e2fsprogs (e.g. e2fsck -V). There was apparently a bug prior to version 1.40.2 that could cause e2fsck (which is called by fsck for EXT2/EXT3 filesystems) to incorrectly report filesystem corruption of this type:
Fix a bug in e2fsck which caused it to incorrectly salvange directories when the last entry's rec_len is bogusly too big. This resulted in a nonsense filesystem corruption to be reported, and required a second run of e2fsck to fully fix up the directory.
The RHEL 4 servers that I have access to have version 1.35 on them, so this could be the real cause of your problem. It might be worth installing a newer version and running fsck again to be sure.
If that doesn't help, it may indeed be a filesystem problem. According to this thread, it appears that this can happen if you have a directory containing a very large number of files (millions or more). "Fixing" it with fsck will result in the inode being truncated, as you saw. In the case of a directory inode, truncation will cause the directory to "lose" files, and those orphaned files will end up in the filesystem's lost+found directory. They will be named after their inode number, not their original names, so although you'll still have the files, it may be difficult to tell them apart.
Best Answer
To over-simplify, an inode is an entry in the filesystem's database. An inode can represent a directory or a file, among other things. If you have run out of inodes, then you likely have too many files on your volume. If you can, delete some files or move them to another volume. Another option might be to tar or compress a bunch of files in to one file.
See https://unix.stackexchange.com/questions/26598/how-can-i-increase-the-number-of-inodes-in-an-ext4-filesystem