When running the following gcloud
command on my existing GKE cluster
SCOPES=(
https://www.googleapis.com/auth/compute
https://www.googleapis.com/auth/devstorage.read_write
https://www.googleapis.com/auth/monitoring.write
https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/monitoring
https://www.googleapis.com/auth/pubsub
https://www.googleapis.com/auth/servicecontrol
https://www.googleapis.com/auth/service.management
https://www.googleapis.com/auth/sqlservice.admin
https://www.googleapis.com/auth/trace.append
https://www.googleapis.com/auth/cloud_debugger
https://www.googleapis.com/auth/cloud-platform
)
gcloud beta container node-pools create $POOL_NAME \
--machine-type $MACHINE_TYPE \
--disk-size $DISK_SIZE \
--enable-autorepair \
--enable-autoscaling \
--min-nodes 1 --max-nodes 4 \
--cluster $CLUSTER \
--zone $ZONE \
--num-nodes 1 \
--scopes $(printf ",%s" "${SCOPES[@]}")
(Note this is regional, so using the beta command – I suspect this doesn't happen with non-regional clusters)
I get the following error:
Creating node pool pool-alpha...done.
ERROR: (gcloud.beta.container.node-pools.create) Operation [<Operation
endTime: u'2018-03-29T08:56:14.989660264Z'
name: u'operation-1522311735033-87b12027'
operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL, 7)
selfLink: u'https://container.googleapis.com/v1beta1/projects/xxxxxxxxx/zones/europe-west1-d/operations/operation-1522311735033-87b12027'
startTime: u'2018-03-29T08:22:15.03391313Z'
status: StatusValueValuesEnum(DONE, 3)
statusMessage: u'All cluster resources were brought up, but the cluster API is reporting that only 0 nodes out of 3 have registered. Cluster may be unhealthy.'
targetLink: u'https://container.googleapis.com/v1beta1/projects/xxxxxxxxxx/zones/europe-west1-d/clusters/digibet-prod/nodePools/pool-alpha'
zone: u'europe-west1-d'>] finished with error: All cluster resources were brought up, but the cluster API is reporting that only 0 nodes out of 3 have registered. Cluster may be unhealthy.
And indeed, the nodes are created but aren't registered on the cluster. A bug with GKE?
Best Answer
Anton Kostenko gave the answer on StackOverflow:
https://stackoverflow.com/questions/49667351/when-creating-a-node-pool-multi-zone-with-gcloud-the-nodes-do-not-register-on
It was a problem with my
--scopes $(printf ",%s" "${SCOPES[@]}")
line which was adding an extra comma at the end which is (somewhat understandably) not validated on the gcloud cli and breaks some internal GKE node registration processI changed it to
--scopes $(IFS=','; echo "${SCOPES[*]}")
(no trailing comma) and now it works