I want to use a kubernetes(cluster-0)with multiple kube-state-metrics to monitor multiple other kubernetes cluster(cluster-1,2,3,4)
In the (cluster-0), I split into multiple namespaces like this:
namespace: monitor
(here I will run prometheus to scrape metrics from other kube-state-metrics pod)
namespaces: cluster-0
(here I will run kube-state-metrics pod to monitor cluster-0
....
namespaces: cluster-4
(here I will run kube-state-metrics pod to monitor cluster-4)
...
And so on…
The problems here is how would I setup prometheus to auto scrape new kube-state-metrics since I will expand the kubernetes cluster very fast, or shrink it. And how to tell the different metrics between them, like the having a header …
I have try using config but just locate single - targets: ['serviceIP:8080']
is not very good in my situation.
Best Answer
As I already mentioned in my comment. What you need is the Prometheus Self Discovery on Kubernetes:
With it you will be able to scrap metrics from new nodes when they come up. More details and examples can be found in the linked docs.