Kubernetes kubelet logging to files and logrotate

kuberneteslog-fileslogginglogrotate

kubelet has an option to store log files in a log-dir (https://github.com/kubernetes/kubernetes/issues/21248).

I start kubelet with the options --logtostderr=false --log-dir=/var/log/kubelet (I have already created the directory /var/log/kubelet) and I indeed see log files with the format kubelet.(hostname).root.log.(date) getting generated in /var/log/kubelet.

In my case I see three files that look like this:

kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.hostname.root.log.WARNING.20180301-152320.161945

Moreover, a symbolic link that points to the latest log file is created for each of the ERROR, INFO and WARNING log files:

kubelet.ERROR -> kubelet.hostname.root.log.ERROR.20180301-152321.161945
kubelet.INFO -> kubelet.hostname.root.log.INFO.20180301-152320.161945
kubelet.WARNING -> kubelet.hostname.root.log.WARNING.20180301-152320.161945

My question is how do I rotate and delete the log files that are older than e.g. 5 days old?
It seems these files are generated by glog, but it also seems that although glog rotates the log files, it doesn't support deleting old log files (https://github.com/google/glog/issues/36).

I am aware of logrotate, but then the files should have a fixed filename in order to be rotated by logrotate.

Is there a way to enable kubelet logging in files with proper log rotation and deletion of old files?

Best Answer

Kubernetes doesn’t provide built-in log rotation, but this functionality is available in many tools.

Basing on the official Logging Architecture:

An important consideration in node-level logging is implementing log rotation, so that logs don't consume all available storage on the node. Kubernetes is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour. You can also set up a container runtime to rotate an application's logs automatically.

As an example, you can find detailed information about how kube-up.sh sets up logging for COS image on GCP in the corresponding configure-helper script.

Related Topic