On our docker implementation on GCE, we are running out of space on the root file system.
Since images themselves are stored on a separate 1TB volume, the images themselves shouldn't be the problem.
One candidate are the centralized logfiles that Docker itself stores (a json file somewhere?), does anyone know where those files/file are/is located, and how we can logrotate/truncate them?
Best Answer
First, I'm using docker 1.1.2 for both client and server, this answer may be obsolete for newer versions of docker as docker evolve quickly.
Location of the file
Find your docker directory. On systems that use apt/debian style system, the package installed by the docker repository
https://get.docker.com/ubuntuuse/var/lib/docker. Chances are that directory is in the same place on other systems (can't confirm).under
containers/**CONTAINER_ID**you'll find infos about the container. In the file**CONTAINER_ID**-json.login that folder, you'll find a file with all the logs for that container. It may look like a json file, it's not. It's a flow a json structures, one per line, each containing one log line (each line ends by a}and the next one starts with a{, thus it's not a valid json as a whole).Example location: -
/var/lib/docker/containers/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6/05b6053c41a2130afd6fc3b158bda4e605b6053c41a2130afd6fc3b158bda4e6-json.logEditing/Altering that file
I suggest you to use that path to see wether or not it's the reason why you're running out of space, but not to log rotate them.
I would rather make sure the container doesn't log too much lines (by using a CMD in the dockerfile that either redirect the output of you process to a file in a volume or to /dev/null - with logs enabled with configuration - and I would then logrotate the log files with another container)