Kubernetes – How to Find Disk Usage on Node

kubernetes

I have a single-node kubernetes cluster and I got a notification that disk usage was over 70%. When I get a shell on the node and run df I see that / is using 75% of the available blocks. I also see a number of entries like this:

overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/47d405a81214ec9b81ccdffae3674260fa2946d30bdd3b398aa1c2d283c8577e/rootfs
overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/569d1fe262f6bf1a59ca99163f231a426d2251ac1beabc47a0393066f48b3e0e/rootfs
overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/32d14863f46e922b7e0b2bbee8a5321c5fabb5811352449d90e486b94161a8cb/rootfs
overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/8c3fa2f9467e792036782e9587f748f846d61fdc23d9afaecc7e7835e6db4f2b/rootfs
overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/626b21093218edcda466ce6d4f31e7b975a0c0473d233fc062ed657499541fc9/rootfs
overlay         51432064 36953456  12350764  75% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c1a7484e1beca4ec1299fc5017002d6fecc9890c86e3246a8bdf53fcc45cf576/rootfs

I have two services that use persistent volumes (not host volumes) and can't think of anything other than docker images that could be using up all of this storage.

How can I figure out what is using this much storage on my kubernetes cluster?

Best Answer

To find out what is using your Node's disk in Kubernetes, you can follow these steps:

  1. SSH into your Node.

  2. Install the ncdu package on your Node using the following command:

    sudo apt-get install ncdu

  3. Run ncdu on the root directory of your Node using the command:

    sudo ncdu /

  4. Wait for ncdu to finish scanning your Node's disk. It will display a list of directories and their respective sizes.

  5. Use the arrow keys to navigate through the directories and subdirectories to find out which one is taking up the most disk space.

  6. Once you have identified the directory that is taking up the most space, you can use kubectl to find out which pod is using it by running the following command:

    kubectl describe pod <pod-name>

    Replace <pod-name> with the name of the pod you want to check.

  7. Look for the Volume section in the output of the kubectl describe pod command. It will show you the PVC (Persistent Volume Claim) that is mounted on the pod.

  8. Use kubectl to describe the PVC using the following command:

    kubectl describe pvc <pvc-name>

    Replace <pvc-name> with the name of the PVC that was mounted on the pod.

  9. Look for the StorageClass and Access Modes sections in the output of the kubectl describe pvc command. It will tell you which StorageClass the PVC is using and how it is being accessed.

  10. Check the StorageClass configuration to see if there are any storage limits or quotas set on it that might be causing disk usage issues.

By following these steps, you should be able to identify which pod is using the most disk space on your Node in Kubernetes.

Related Topic