Linux – Disk full on linux server, blocks used is much less then blocks avalable

duhard drivelinux

Output of df is:

[root@backup log]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGro  1889811408 1861658948         0 100% /
/dev/sda1               101086     16235     79632  17% /boot
tmpfs                  1815760         0   1815760   0% /dev/shm

So available block should be 28.152.460, but it's 0. I've been deleting a crap load of files, and used blocks is going down, but available remains at 0.

Output of df -i is: 

[root@backup log]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGro 487751680 238360803 249390877   49% /
/dev/sda1              26104      37   26067    1% /boot
tmpfs                 219784       1  219783    1% /dev/shm

So it's not a lack of inodes.

Output of lsof +L1 is:

[root@backup log]# /usr/sbin/lsof +L1
COMMAND     PID  USER   FD   TYPE DEVICE SIZE NLINK      NODE NAME
mysqld     2444 mysql    4u   REG  253,0    0     0 268795908 /tmp/ibSlaKC7 (deleted)
mysqld     2444 mysql    5u   REG  253,0    0     0 268795909 /tmp/ibhFuyGr (deleted)
mysqld     2444 mysql    6u   REG  253,0    0     0 268795910 /tmp/ibbNinKL (deleted)
mysqld     2444 mysql    7u   REG  253,0    0     0 268795911 /tmp/ibz1ia55 (deleted)
mysqld     2444 mysql   11u   REG  253,0    0     0 268795912 /tmp/ibM3IHvr (deleted)
crond      2549  root    3u   REG  253,0    5     0 248579098 /var/run/crond.pid (deleted)
yum-updat  2620  root   14w   REG  253,0    0     0 248611115 /var/run/yum.pid (deleted)
ssh       16256  root    0u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    1u   CHR  136,0          0         2 /dev/pts/0 (deleted)
ssh       16256  root    2u   CHR  136,0          0         2 /dev/pts/0 (deleted)

I can't run 'du' because 99% of the disk usage is under /var/backups which contains probably ~100 million files (some idiot decided to rsync code from live server with subversion directories, so it's a lot of tiny files), so running 'du' would take days or weeks.

Does anyone have any suggestions on how to proceed?

Best Answer

If this is an ext filesystem, the default root reserved space would be 5% of 1889811408 blocks, or 94490570 blocks. In other words, you have about 66GB more to delete before df will report free space available.

Use tune2fs -m 1 /dev/mapper/VolGro to decrease the reserved amount to 1%, or -r NNNN to set it to a specific number of blocks. There needs to be enough reserved space that logging can continue even after users have "filled" the disk (though if you're filling the disk as root, this won't save you from problems when the drive is absolutely full)

Other filesystems probably have reserved blocks as well, but the commands to adjust these will differ.

Related Topic