GKE Clusters not coming back after project restore

google-cloud-platformgoogle-kubernetes-engine

I deleted my GCP project (whoops!) and then restored it about 30 minutes later. GCE VM instances did not start automatically, but I started them. I also had to manually enable the compute API for some reason (gcloud services enable compute.googleapis.com).

My Kubernetes clusters appear in GKE but they are in error state and just say "The cluster has a problem". When I click the "Nodes" tab I see "Could not get data from the cluster (unavailable)"

If I check the describe status of my cluster (gcloud container clusters describe cluster_name) then I see a status of ERROR. Attempting to do anything to the cluster, like upgrade it, causes this:

ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400, message=Cluster is currently being created, deleted, updated or repaired and cannot be updated.

Is there any way to get these clusters back online without manually recreating and repopulating them?

Best Answer

Went through the GCP support channel and they were able to get my cluster back up and running. According to the support rep, this is not always necessary:

Please note that it's not the expected behavior for the clusters to be in a problematic state after a project restore. It can happen from time-to-time though, but are rare occurrences.

So that did it for me, not sure what you would need to do if you didn't have a GCP support package though.