Linux ext3 Directory index full – internals

ext3kernellinux

Encountered a situation today on a server that has me wondering. Here's the scenario:

Syslog shows:

kernel: EXT3-fs warning (device sdb2): ext3_dx_add_entry: Directory index full!

Found the culprit to be a directory with 9.1 million files in it. I know it was 9 million files because I used this to delete them:

perl -e 'my $i=0;for(<*>){$i++;((stat)[9]<(unlink))} print "Files deleted: $i\n"'

Right after completion, I ran ls – that took about 3 minutes, and returned 1 file.

A few minutes later, a fresh batch – again 9.1 million files have appeared in the same directory, and syslog showed again:

kernel: EXT3-fs warning (device sdb2): ext3_dx_add_entry: Directory index full!

I ran the delete again, and the exact same scenario repeated itself. Few minutes later, a new batch of over 9 million files.

The files that just appeared are old (about 3 months old).

Can someone confirm if this is the expected behavior of ext3?

  • Directory index full is raised, well, when it's full
  • New files are allowed to be created, but can't be added to the index
  • New files are cached "somewhere"
  • Once a slot is freed, the new file is added to the index (and hence will show up with e.g. ls)

I suspect that this is what's happening, but I currently don't have any proof.

Any feedback appreciated!

Please note the question isn't about how to fix it, it's about understanding what's happening here.

Best Answer

"...the question isn't about how to fix it, it's about understanding what's happening here...."

My guess is that you're suffering of a (severe?) file-system corruption and... the more files are going to be (virtually?) created and removed, the more severe the corruption is going to be.

I'm saying this 'cause:

  1. you wrote "...A few minutes later [...] again 9.1 million files have appeared in the same directory...". Let's assume that with "few minutes" you intended 15 minutes. This means roughly 10K files created per second. Even tough your server/storage can handle such a load, it's hard to believe that while such a creation process is running there are not any signs of such an activity! So, at least, the double fact that:

    • 10K files per seconds are created (in a 15 minutes timeframe);
    • you're not noticing any strange load/behaviour of your system

    let me think that... the creation process is "fake" and, as such, you have file-system integrity issues;

  2. even tough the message "ext3_dx_add_entry: Directory index full!" let me think that in such a scenario it should be not possible to create additional files within related filesystem (BTW: as mentioned by @Stefan , commenting your OP), you seem to be able to effectively create additional files. That's again -- IMHO -- another fact raising attention. As a partial mitigation, the word "warning" (...and not "error") in the log messages (EXT3-fs warning).... makes me think that even kernel-developers expected that such a situation is not so "critical" (...they think it's a warning... and not an error. So it should not be something so... terrible!);

In addition to the above, I also found this other SF post confirming that, at least is that specific situation, problem was a file-system-integrity issue.

As for the final part of your question (the one related to the "caching" hypothesis), even if I'm not at all a kernel-hacker, I strongly believe that this is not a kernel behaviour as it would be out of scope of the kernel, being something to be approached cross-module (not being possible to implement dealing only with the ext3 module). But, please, don't blame me if this last sentence is absolutely wrong! It's only my "feeling" :-)

Update

As for point 2) above, I was wrong: the "Directory Index" seems not to be strictly related to the file-system structure used to effectively store both file-data and file-metadata. Instead it's "only" a means to optimize searching in directories containing lots of files [see here or this other SF post here]. This seems to explain why the log reports a "warning" (and not an "error") as, in my opinion, when "Directory Index" is full... everything can proceed as usual, without the benefits of the indexing.

Related Topic