Ubuntu Server – How to Fix Disk Space Slowly Filling Up

disk-space-utilizationUbuntu

We had our samba server (ubuntu 8.04 ltr) share fill up the other day but when I went to look at it I cant see any of the shares have to much on them

we have 5 group shares
and then each users has an individual share

one users has 22gigs of stuff a few others have 10-20mb of stuff and everyone else is empty

so maybe like 26gigs total

I deleted a few files yesterday and freed up about 250mb of space
today when i checked it it was completely full again and i deleted some older files and freed up about 170mb of stuff but i can watch it slowly creep down in free space.

I keep running a df -h

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            241690180 229340500    169200 100% /
varrun                  257632       260    257372   1% /var/run
varlock                 257632         0    257632   0% /var/lock
udev                    257632        72    257560   1% /dev
devshm                  257632        52    257580   1% /dev/shm
lrm                     257632     40000    217632  16% /lib/modules/2.6.24-28-generic

/volatile

what can I do to try to hunt down whats taking up so much of my hdd?
(im fairly new to unix in general so i apologize if this is not well explained)

Best Answer

(This is a Linux focused answer. Other UNIX variants may be different.)

There are two pieces of information relevant to your problem: (1) which files are filling up your file system, and (2) which processes are writing to those files.

Notes

Below, when I put the $ character in commands, that's probably a place holder where you need to substitute a real value. Hopefully, it's obvious where to do that and where not to.

Which Files?

Be aware that there are really two resources in most file system types that can be used up by individual files: meta-data (e.g. inodes), and real data. You can see the number of inodes (search Google for a definition, but they're "pointers" to the structures that make up your files) with a command like:

df -i

... and as you already know, something like this will show the space being used by real data:

df -h

Also, be aware that file system space can be taken up by files that don't exist on disk. These files are still in the open state by some process, but have been removed (we'll cover that below).

Once you've identified the full file system(s), then you need to start looking for lots of little files, a few big files, or both. Running out of meta-data resources is usually caused by having lots of little files, whereas running out of real data resources is usually caused by a few big files. I like to use this command to help find the big files:

sudo find $file_system -mount -ls | awk '{print $7, $11}' | sort -rn > $output

... and this command to help find directories with lots of little files (Update:: added null termination to improve file name handling):

sudo find . -mount -print0 | xargs -0n 1 dirname | sort | uniq -c | sort -rn > $output

... be aware that these commands may take a while to run, and do a lot of I/O, depending. Once run, you can read through $output to find the offending files or directories. The name and location of each may give you a hint about where the data's coming from, but requires some Linux experience.

Once you've identified the offenders, you can rm $file to get rid of the problem.

Which Processes?

The most straight forward way to find the processes potentially filling up your file system is to run a command like:

fuser -c $file_system 2>/dev/null

... which will tell you the PID of processes that have open file descriptors (files and network sockets) for the given file system (the 2>/dev/null part gets rid of some information you don't need). You might be able to deduce just from these PIDs which process is filling up your file system. Search for the processes with:

ps -ef | grep $pid

You can also try running this command which will give you even more detail (and help identify open files with no corresponding file name on the disk -- I mentioned this above):

sudo lsof $file_system | grep $directory_filling_up

... and if you've identified a suspect PID from the fuser command, you can do this:

sudo lsof -p $pid

The problem with fuser and lsof is that they only give you a snapshot of the system at the time your run the command. If the offending process doesn't happen to be writing when you run them, you're out of luck. You can counter this by repeatedly running them over time and saving the output. This will require reading through the output to find patterns, or writing a program to do it for you. An alternative is to use a tool like SystemTap. SystemTap allows you to trap all kinds of useful information, and is "programmable." It even comes with some sample source files that would allow you see what processes are writing to what files over some span of time. It would be perfect, but it's an advanced tool and requires a lot of Linux knowledge.

Once you've identified the offending process(es), you can kill (and maybe restart them). If the process is associated with the operating system or some well packaged software, there will probably be a mechanism for restarting them, but it will depend on your Linux distro (I think Ubuntu will allow you to run something like /etc/init.d/$init_script restart, but you'll have to check your distro's documentation). Otherwise, you can kill it with kill $pid or kill -9 $pid if it's not behaving. Be careful to note how the process was running (e.g. what were it's arguments shown in ps -ef) in case you need to restart it (you may need to refer to that software's documentation).

Related Topic