I run pods in kubernetes (1.17.0) that write logs to stdout. This pods are going to be up for a long time. I use promtail to gather logs and send to loki storage. So I can configure to store logs, for example, for last 7 days.
However, I don't clearly understand how kubernetes tracks these logs? I know that kubernetes deletes logs when pod is evicted, but my pods will not be evicted. That means kubernetes will store all logs and eventially node's SSD will be full? How can I configure kubernetes to delete old logs?
Best Answer
Looking at the Logging architecture from the official k8s documentation there are some options:
First one is logging at node level:
Second one is a sidecar container with logging agent:
Alternatively you can always setup your log retention policy at docker level by specifying appropriate flags. Check out docker docs for that.