My yaml files for various deployments, services and ingresses have created a working setup before. I tried fixing a small issue with an ingress and can't get back to a working state.
My pods start without error and the services are up and running. The logs have no health check entries so the issue seems to be in getting the services to be contacted by name. When the ingress in creating, I see this event in the GKE interface (and not when using kubectl describe):
Error during sync: error while evaluating the ingress spec: could not find service "default/web";
The service is indeed up:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 5h
kubernetes ClusterIP 10.31.x.x <none> 443/TCP 5h
redis-master ClusterIP 10.31.x.x <none> 6379/TCP 5h
web NodePort 10.31.x.x <none> 80:30110/TCP,443:31404/TCP 5h
websocket NodePort 10.31.x.x <none> 28080:30186/TCP 5h
I've tried restarting services and changing to an nginx ingress. My only remaining idea is to nuke the cluster. It's as if the k8s internal DNS is broken.
Best Answer
I had both port 80 and 443 pointing to 3000 for the web service where my Rails app was running. I enabled
force_ssl
in the Rails configuration. I figured all this would do is redirect http requests to https, but it turns out this resets the port you're serving on. You need to specify the port if you want to enable force_ssl: https://github.com/rails/rails/issues/19820Because port 3000 was no longer responding with 200, that service was marked unhealthy, despite nothing otherwise being wrong with it.