Linux – disk space missing

disk-space-utilizationhard drivelinux

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
rootfs                9.9G  7.2G  2.2G  77% /

# du -hx --max-depth=0 /
3.2G    /

As you can see, df says 7.2GB is used, but du can only find 3.2GB of it. The server has been rebooted since I noticed this, so it's not a deleted file. Additionally, lsof doesn't show me anything interesting. What else could it be?

Best Answer

I was having the exact same issue on my ext4 system and just wanted to post my solution for future reference. When my drive initially filled out, I deleted a bunch of logs out of /var/log. That cleared up a couple GB, but within a few days I ran out of space again, and du -h and "mount --bind / /mnt" did not point to the culprit. What finally got it was when I ran lsof.

lsof
...
rsyslogd   1766      root    2w      REG                9,1   12672375940     264014 /var/log/messages (deleted)
...

When I had deleted the messages log file, the rsyslog service still kept it open, but hidden. Running a "touch /var/log/messages;service rsyslog restart" cleared up the problem and my disk space was reclaimed.

The lsof output can be a bit overwhelming, especially if you have a busy system (it was over 1000 lines long on mine). If you grep for "deleted" in the lsof output, it should help to pinpoint the problem process.

Related Topic