Stackdriver missing GKE Logs

• Need to make sure the cluster is set up with Stackdriver logging/monitoring enabled.
• Following the mentioned documentation, you should have a pod named counter. If you run:

$ kubectl logs counter # Are you able to see the output for that pod?

• If you intend to customize Stackdriver Logs for Kubernetes Engine with Fluentd, this tutorial suggests to first create the cluster service account and assign required roles. Next, the cluster is created with "no-enable-cloud-logging"; and, deploying the Fluentd daemonset.

• Trying the advanced filter like below should help:

resource.type="container"

resource.labels.pod_id="es-data-0"

The Master node and the worker nodes are each in different networks, the master is in a Google managed one and the worker nodes are in your VPC. With a standard cluster, the master communicates with the nodes via external IP. With a private cluster, the Master node and the worker nodes are connected via a network peering and communicate via internal IPs.

This causes problems when connecting to the master directly over other peered networks or VPN connections because the network peering routes to the master are not propagated over VPNs and network peering.

For your use case, disable the external master endpoint. Once this is done, when you run the get-credentials command, your kube config will have the internal master endpoint instead of the external one. You will then need to connect to your master (kubectl) from the VPC network (a bastion host or a proxy).

Instead, I recommend leaving the external end point active, use get-credentials without using --internal-ip so that your kube config will use the external end point and thus you can connect from anywhere. To make sure that you master is secure, use Master Authorized Networks to define the external IPs or CIDRs you will be connecting from

I am fairly certain the kubectl exec and logs commands are failing because of how you are getting the credentials.

One last thing worth checking as well, GKE creates firewall rules and routes automatically (they will be called gke-...), these are required to ensure that SSH tunnels from the master to the nodes work properly.