Disk usage metrics missing from Prometheus node exporter

disk-space-utilizationhard drivemetricsmonitoringprometheus

I used Prometheus and node exporter a while ago and had access to node_filesystem_* metrics to monitor disk usage but I've recently fired it up on some other servers (Ubuntu Linux) and those metrics seem to be missing.

According to this https://github.com/prometheus/node_exporter those metrics should be enabled by default and available from Linux hosts. What might be causing them to not appear?

To clarify, I can get metrics for CPU, memory etc so it's not that node exporter just isn't working.

Best Answer

Pay your attention to /etc/default/prometheus-node-exporter: for e. g., recently I realised Ubuntu's (18.04) package cuts out by default all disks statistics:

ARGS="--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$ \

Pretty weird as to me, but at least can be adjusted.

Related Solutions

How to monitor memory usage of linux server using nagios

i suggest you rather monitor swap usage. check out check_swap plugin - it comes by default [at least in debian].

How to get CPU usage percentage for a namespace from Prometheus

I found out why I couldn't use the metric I cited above. It's because usually there are only a few pods that even have a CPU limit setting. It is not needed in general and it would make the cluster clumsy.

sum(kube_pod_container_resource_limits{resource="cpu", unit="core", namespace="$Namespace"})

does sum all the existing limits on the pods of the namespace but that's not the theoretical 100% CPU usage of the namespace. This is why percentages over 100% appear sometimes.

However, I learned that theoretically the namespace could use up all the resources delegated to the nodes of the cluster. I also learned that our product would likely run on machines very similar to this test server in production. So to get the CPU usage as a percentage, it is valid to calculate namespace CPU usage / available CPU in cluster in my lucky case.

Here is how I do that:

sum (rate (container_cpu_usage_seconds_total{namespace="$Namespace"}[1m])) / sum(machine_cpu_cores) * 100

where $Namespace is the name of the namespace.

(The same applies to memory usage.)

So this is what I'm going to monitor while running load and stress tests.

Best Answer

Related Solutions

How to monitor memory usage of linux server using nagios

How to get CPU usage percentage for a namespace from Prometheus

Related Topic