Cryptic Stackdriver HTTP Uptime check error using Google Cloud Platform

health checks managed instance groups VS health checks load balancing

The health checks used by managed instance groups are the same health checks used by load balancing, with some differences in behavior. Health checks that you apply to load balancing services help a load balancer determine where to direct network traffic. These health checks do not cause Compute Engine to recreate instances. Health checks that you apply to managed instance groups will proactively signal to the managed instance group to delete and recreate instances if they become UNHEALTHY.

For the majority of scenarios, use separate health checks for load balancing and for monitoring managed instance groups. Health checking for load balancing can and should be more aggressive since these health checks determine whether an instance receives user traffic. Since customers might rely on your services, you want to catch non-responsive instances quickly so you can redirect traffic if necessary. In contrast, health checking for instance groups will cause Compute Engine proactively replace failing instances so you could create health checks that are more conservative than health checks for a load balancer.

https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances#monitoring_groups

You can look directly at the API reporting graphs to see that your calls to google.devtools.cloudtrace.v1.TraceService.PatchTraces fail with 403 Forbidden errors.

403 Forbidden is caused when a request to a server is not authorized. Therefore, your calls from your Container Engine (GKE) cluster to 'cloudtrace.PatchTraces' are not authorized.

This could be due to the limited Stackdriver Trace 'Write Only' permission you set. Also, ensure that you added the 'trace.append' scope when creating the cluster.

Concerning the missing logs in Stackdriver. Once Stackdriver Logging is enabled for your cluster you should be able to simply write to 'STDOUT' and 'STDERR'. Fluentd should take care of sending this to Stackdriver and the output should be available in the Log Viewer under the 'GKE Container' dropdown.

Alternatively you can always use the Stackdriver Logging Client Library to directly write to Stackdriver, just as you are doing with the Stackdriver Trace API. You should also set the Log Severity to make filtering easier in the Log Viewer to find specific error logs (like the 403s you are seeing).