Ext4 listing of files very slow in one specific directory that contained lots of files before

debuggingext4filesystemsperformance

Background

I had a small logrotate misshap… Logrotate would rotate the archived logs by misstake causing a quadratic growth of files in my /var/log/. And by the time I caught wind that something was awry, /var/log/ already contained a few million files…

I managed to (after some hairloss and find/sed/grep magic) remove all offending files and fix my logrotate config. And thought all was well…

Problem

Whenever I ls/du -hs or otherwise list the contents of /var/log/ (which now contains 80mb of archives/logs and at most a few hundred files) the process doing that hangs for a good minute or two. I do believe this is somehow related to the logrotate misshap but I'm not certain, it could be something else. Anyway I'm at loss to where to start debugging or looking for a fix for this. Please help :3

Other info

uname -a
Linux xxx 3.3.8-gentoo #18 SMP Sat Sep 21 22:44:40 CEST 2013 x86_64 Intel(R) 
Core(TM)2 CPU 4400 @ 2.00GHz GenuineIntel GNU/Linux

cat /proc/meminfo 
MemTotal:        2051552 kB
MemFree:           75612 kB
Buffers:            9016 kB
Cached:          1740608 kB
SwapCached:            0 kB

CFQ IO scheduler + SLUB allocator

I thought this: How many files in a directory is too many? (Downloading data from net) was related but I don't have the files left anymore.

Edit

The problem persists even after a call to init 1 so I think it is safe to assume there is no other process to blame but the FS.

Solution (as applied from accepted answer)

init 1
mv /var/log /var/log1
mkdir /var/log
chmod --reference=/var/log1 /var/log
chown --reference=/var/log1 /var/log
tar -C /var/log1 -cvp . | tar -C /var/log -xvp
rm -rf /var/log1
init 5

Best Answer

Directories only ever grow in size, not shrink. Try moving all those files out into a a temporary directory (like log2) then rmdir the old directory and rename the temp one as the new permanent one.

Related Topic