Can’t see pod logs in Stackdriver UI for cluster deployed on GKE

elasticsearchgoogle-kubernetes-enginegoogle-stackdriverkubernetes

I deployed an Elasticsearch cluster on Google Kubernetes Engine.
I am able to view logs of specific pods with the 'kubectl logs' command from my PC.

>kubectl logs es-data-0
...
[2018-06-18T07:35:11,220][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4510405] overhead, spent [290ms] collecting in the last [1s]
[2018-06-18T09:48:19,194][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4518376] overhead, spent [366ms] collecting in the last [1.1s]
[2018-06-18T10:01:49,344][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4519183] overhead, spent [641ms] collecting in the last [1s]

I followed this tutorial to deploy a ConfigMap & DaemonSet for Stackdriver logging:
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/

the fluentd pods are running:

>kubectl get pods
NAME READY STATUS RESTARTS AGE
cerebro-59648dc47c-vr964 1/1 Running 0 25d
es-client-7bff44b8f5-2wqcs 1/1 Running 0 12d
es-client-7bff44b8f5-vnrhg 1/1 Running 0 12d
es-data-0 1/1 Running 0 52d
es-data-1 1/1 Running 0 52d
es-data-2 1/1 Running 0 52d
es-master-6bf767f949-8fpjl 1/1 Running 0 52d
es-master-6bf767f949-brjpq 1/1 Running 0 52d
es-master-6bf767f949-gx2jp 1/1 Running 0 52d
fluentd-gcp-v2.0-7mncl 1/1 Running 0 43m
fluentd-gcp-v2.0-rsfmc 1/1 Running 0 43m
fluentd-gcp-v2.0-tbh9t 1/1 Running 0 43m
kibana-595858b4b7-5npcr 1/1 Running 0 52d
nginx-ingress-controller-86c8447687-z4rjq 1/1 Running 2 52d
nginx-ingress-default-backend-6664bc64c9-q2hnm 1/1 Running 338 52d
>kubectl get ds --all-namespaces
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
default fluentd-gcp-v2.0 3 3 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 44m
kube-system fluentd-gcp-v2.0.10 3 3 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 52d
kube-system metadata-proxy-v0.1 0 0 0 0 0 beta.kubernetes.io/metadata-proxy-ready=true 52d
kube-system nvidia-gpu-device-plugin 0 0 0 0 0 <none> 52d

The ConfigMap is present:

>kubectl get cm
NAME                              DATA      AGE
cerebro-configmap                 1         52d
curator-config                    2         6d
fluentd-gcp-config                4         47m
ingress-controller-leader-nginx   0         52d
nginx-ingress-controller          1         52d

Finally, I proceeded to Logs Viewer Page, where I selected "GKE Container":
https://console.cloud.google.com/logs/viewer

Unfortunately, I do not see any logs from any pods. As mentioned previously, when using "kubcetl logs" command I can view logs from all the pods from my PC.

Best Answer

  • Need to make sure the cluster is set up with Stackdriver logging/monitoring enabled.
  • Following the mentioned documentation, you should have a pod named counter. If you run:

$ kubectl logs counter # Are you able to see the output for that pod?

  • If you intend to customize Stackdriver Logs for Kubernetes Engine with Fluentd, this tutorial suggests to first create the cluster service account and assign required roles. Next, the cluster is created with "no-enable-cloud-logging"; and, deploying the Fluentd daemonset.

  • Trying the advanced filter like below should help:

resource.type="container"

resource.labels.pod_id="es-data-0"

Related Topic