Ubuntu – OSSEC disk space usage

You need to drill further down to see what's using all the space. Applications you run store data in subfolders of that folder. Locating the particular subfolder that's eating up all the disk space will help pinpoint the application.

An aside: A neat little freeware utility that I like to use to show subfolder sizes, and one that I would use in this instance, is TreeSize Free, available from: http://www.jam-software.com/freeware/index.shtml

I think you have something writing to a file that has been deleted from the drive but not yet closed by the application/server, so the space remains allocated on disk but can't be seen by du since the file was removed from the filesystem. The lsof program lists processes that have files open. If you had more filesystems mounted and the number didn't fluctuate so much, then I would have suggested that you had a filesystem mounted on top of a directory that wasn't empty (though you could try umount /var/lib/ureadahead/debugfs and make sure that directory is empty and there isn't a bunch of junk written to the directory hiding under that mountpoint).

If this is the case, then you should easily find these with sudo lsof | grep deleted. lsof includes (deleted) in the last column if a file has been deleted while a process still has it open. The first column is the name of the command, the second column is the PID. You can get a more detailed look at the command using ps for instance ps auxww | grep PID, or ps auxwwf | less -S to see the process list in "forest" mode so you can see what process that PID came from. Once you've tracked down the process(es) that are holding open giant files, you can stop it to free the drive space, then figure out how to fix it to close the file properly. The usual cause of this is a logrotate script that renames/deletes log files but doesn't notify the application that it has done so (either through an appropriate signal with kill or by restarting the application), so the application continues to hold the old log file open.