Linux – Inode usage gone up to 100% hence can’t uninstall any package

aptinodelinuxlinux-kernelubuntu-14.04

My server's inode usage has gone up to 100% due to which I am not able to uninstall extra linux-kernel-headers and linux-kernel-images

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/xvda1     524288 523970    318  100% /
none           481856      2 481854    1% /sys/fs/cgroup
udev           480561    396 480165    1% /dev
tmpfs          481856    317 481539    1% /run
none           481856      1 481855    1% /run/lock
none           481856      1 481855    1% /run/shm
none           481856      3 481853    1% /run/user
/dev/xvdb      262144     11 262133    1% /mnt

Also HDD usage are up to only 78%

$ sudo df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  5.7G  1.7G  78% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            1.9G   12K  1.9G   1% /dev
tmpfs           377M  368K  377M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G     0  1.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb       3.9G  8.1M  3.7G   1% /mnt

When I try to remove headers and images using sudo apt-get autoremove command, It gives me below error.

$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-headers-3.13.0-83-generic : Depends: linux-headers-3.13.0-83 but it is not installed
E: Unmet dependencies. Try using -f.

When I do sudo apt-get -f install, It gives me below error.

dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h'): No space left on device
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

When I do $ sudo dpkg --configure -a, It gives me below error.

    libpostfix-dns.so.1 -> libpostfix-dns.so.1.0.1
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: No space left on device
dpkg: error processing package libc-bin (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: error: unable to create new file '/var/lib/dpkg/status-new': No space left on device

Now I am not sure how to tackle this issue. Any experts advice would be a great input. I am not sure from where should I start deleting files. Here is description of filesystem.

/$ sudo du -sh *
9.6M    bin
418M    boot
12K dev
6.9M    etc
216K    home
0   initrd.img
0   initrd.img.old
789M    lib
4.0K    lib64
16K lost+found
4.0K    media
28K mnt
684M    opt
du: cannot access ‘proc/21817/task/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/task/21817/fdinfo/4’: No such file or directory
du: cannot access ‘proc/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/fdinfo/4’: No such file or directory
0   proc
23M root
372K    run
9.4M    sbin
4.0K    srv
0   sys
19M tmp
2.9G    usr
669M    var
0   vmlinuz
0   vmlinuz.old

Note: I have already deleted log files older than 20 days. It didn't help to minimise inode usage and I am stuck now.

Best Answer

Given your df -i output and seeing that you are not using any LVM - the only solution you have is to start deleting files.
Every file present on the system / partition will consume one inode.
Directories also consume inode.
So you need to figure out what's driving this number of files up.
Perhaps it's some process creating many temp files.
Perhaps it's log files.

You can use below command to see which directory is the biggest offender

find / -printf "%h\n"|grep -v "^/proc"|grep -v "^/sys"|cut -d\/ -f1-3|sort|uniq -c|sort -rn
Related Topic