I'm in the process of setting up our Kubernetes infrastructure. Our GKE cluster is up and running. I have successfully deployed a test service which is reachable and behaves as it should.
The test service logs a message on startup and every time it receives a request but those messages don't show up in the Cloud Logging UI.
I know it's not the container image because it works just fine when I run it locally. There must be some issue that prevents the container logs from reaching Cloud Logging.
The cluster Cloud Logging option is set to System, Workloads and the node pool service account has the logging.logWriter role.
At this point I'm out of ideas on what the problem could be. I found several older post describing similar problems but those mostly had to do with the image itself or migrating from Legacy Stackdriver which is not the case here.
Best Answer
I had the same issue as well. Enabling Cloud Logging to System and Workloads does not seems to get it fixed as well. I opened a GCP case and it magically worked. Not sure if there's something in the background which they did to fix.