GCE Health Check settings are reverting to default

google-cloud-platformgoogle-compute-enginegoogle-kubernetes-engine

I set up a Load Balancer on the GCP using an Ingress in the Kubernetes Engine.

The Backend Service by default registered all instances as unhealthy. I updated the health check to the HTTPS protocol and the /healthz url, since that's the url for my health check.

I leave it alone for a couple of minutes and the backend service states all instances are healthy now, then I see in the logs that my pod is serving 200 responses at the /healthz.

I navigate to the page, and it starts showing a 502 error. I go back to the health check page in the console, and the health check reverted to HTTP at /.

What caused the health check to revert back to HTTP? Here's the logs showing that for 10 mins the health check continues to request the https I set, then the request changes back to http.

...
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.14', 48346): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:02:35 iostream:1451] SSL Error on 9 ('10.128.0.13', 63030): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.75ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.13) 0.80ms
[I 180621 20:02:43 web:2106] 200 GET /healthz (10.128.0.14) 0.74ms
...
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.13) 1.26ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.14) 0.63ms
[I 180621 20:05:46 web:2106] 200 GET /healthz (10.128.0.15) 0.64ms
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.14', 49971): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:46 iostream:1451] SSL Error on 9 ('10.128.0.15', 62893): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 52191): [SSL: HTTP_REQUEST] http request (_ssl.c:833)
[W 180621 20:05:48 iostream:1451] SSL Error on 9 ('10.128.0.13', 60549): [SSL: HTTP_REQUEST] http request (_ssl.c:833)

EDIT: I'm adding the readiness probe for the pod in question, that the ingress is attached to.

readinessProbe:
  httpGet:
    port: 8902
    scheme: HTTPS
    path: /healthz
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1

Best Answer

I checked and it shouds an intended behaviour.

Kubernetes controls the health check and configures it from data inside the kubernetes cluster. Kubernetes does not see any changes made to the health check via gcloud.

Therefore I guess it is normal that it checks it and any modification get reverted.

The solution to use readinessProbe instead.

Please take a look also to this:

Health checks

Currently, all service backends must satisfy either of the following requirements to pass the HTTP(S) health checks sent to it from the GCE loadbalancer:

  1. Respond with a 200 on '/'. The content does not matter.

  2. Expose an arbitrary URL as a readiness probe on the pods backing the Service.

The Ingress controller looks for a compatible readiness probe first, if it finds one, it adopts it as the GCE loadbalancer's HTTP(S) health check. If there's no readiness probe, or the readiness probe requires special HTTP headers, the Ingress controller points the GCE loadbalancer's HTTP health check at '/'.

Related Topic