Domain – determining disk space usage on plesk domain

(This is a Linux focused answer. Other UNIX variants may be different.)

There are two pieces of information relevant to your problem: (1) which files are filling up your file system, and (2) which processes are writing to those files.

Notes

Below, when I put the $ character in commands, that's probably a place holder where you need to substitute a real value. Hopefully, it's obvious where to do that and where not to.

Which Files?

Be aware that there are really two resources in most file system types that can be used up by individual files: meta-data (e.g. inodes), and real data. You can see the number of inodes (search Google for a definition, but they're "pointers" to the structures that make up your files) with a command like:

df -i

... and as you already know, something like this will show the space being used by real data:

df -h

Also, be aware that file system space can be taken up by files that don't exist on disk. These files are still in the open state by some process, but have been removed (we'll cover that below).

Once you've identified the full file system(s), then you need to start looking for lots of little files, a few big files, or both. Running out of meta-data resources is usually caused by having lots of little files, whereas running out of real data resources is usually caused by a few big files. I like to use this command to help find the big files:

sudo find $file_system -mount -ls | awk '{print $7, $11}' | sort -rn > $output

... and this command to help find directories with lots of little files (Update:: added null termination to improve file name handling):

sudo find . -mount -print0 | xargs -0n 1 dirname | sort | uniq -c | sort -rn > $output

... be aware that these commands may take a while to run, and do a lot of I/O, depending. Once run, you can read through $output to find the offending files or directories. The name and location of each may give you a hint about where the data's coming from, but requires some Linux experience.

Once you've identified the offenders, you can rm $file to get rid of the problem.

Which Processes?

The most straight forward way to find the processes potentially filling up your file system is to run a command like:

fuser -c $file_system 2>/dev/null

... which will tell you the PID of processes that have open file descriptors (files and network sockets) for the given file system (the 2>/dev/null part gets rid of some information you don't need). You might be able to deduce just from these PIDs which process is filling up your file system. Search for the processes with:

ps -ef | grep $pid

You can also try running this command which will give you even more detail (and help identify open files with no corresponding file name on the disk -- I mentioned this above):

sudo lsof $file_system | grep $directory_filling_up

... and if you've identified a suspect PID from the fuser command, you can do this:

sudo lsof -p $pid

The problem with fuser and lsof is that they only give you a snapshot of the system at the time your run the command. If the offending process doesn't happen to be writing when you run them, you're out of luck. You can counter this by repeatedly running them over time and saving the output. This will require reading through the output to find patterns, or writing a program to do it for you. An alternative is to use a tool like SystemTap. SystemTap allows you to trap all kinds of useful information, and is "programmable." It even comes with some sample source files that would allow you see what processes are writing to what files over some span of time. It would be perfect, but it's an advanced tool and requires a lot of Linux knowledge.

Once you've identified the offending process(es), you can kill (and maybe restart them). If the process is associated with the operating system or some well packaged software, there will probably be a mechanism for restarting them, but it will depend on your Linux distro (I think Ubuntu will allow you to run something like /etc/init.d/$init_script restart, but you'll have to check your distro's documentation). Otherwise, you can kill it with kill $pid or kill -9 $pid if it's not behaving. Be careful to note how the process was running (e.g. what were it's arguments shown in ps -ef) in case you need to restart it (you may need to refer to that software's documentation).

I'd suggest breaking the data up into multiple partitions if that is at all feasible. No matter what tools you use, scanning through that many files is going to take time. If it were on separate partitions, you could at least narrow the problem down to a single partition first. But that may not be an option for what you are doing.

du is likely the best tool for what you are looking for. Here's how I use it:

If your directory structure looks like:

/mount/1/abc/123/456/789, 
/mount/1/def/stuff/morestuff/evenmorestuff
/mount/2/qwer/wer/erty

I would run:

du -s /mount/*/* | sort -n

That will give you the total usage of each second level directory, sorted by size. If it takes a long time to run, direct it to a file and run it overnight.

Your output will look like this:

10000 /mount/1/abc
20000 /mount/1/def
23452 /mount/2/qwer

Then you just hope that breaks it down enough to see where the problem spots are.

If this is a regular issue, you could have it run that command every night at a time when your system isn't as busy and save the output to a file. Then you immediately have some recent data to look at when you notice the problem.

One other option you may wish to look at is quotas - if this is shared storage and they are all using different user accounts, setting very high quotas might work to prevent runaway processes from using gobs of storage space.