Google cloud platform: k8s master cannot ssh to nodes

You can check if the instance is open to connections on port 22 by running sudo nmap -Pn -p 22 <instance IP>.

If you have port 22 open in the instance you should check the serial console output for errors in sshd.

If there are no errors, try testing connectivity to the instance by using telnet IP 22. You should get the SSH prompt. When testing try the internal IP and the external one (if you have one). If there is no SSH prompt, this would indicate that the service is down in the other instances. You might also try connecting to other VMs using the browser or the serial console to verify the service status.

The Master node and the worker nodes are each in different networks, the master is in a Google managed one and the worker nodes are in your VPC. With a standard cluster, the master communicates with the nodes via external IP. With a private cluster, the Master node and the worker nodes are connected via a network peering and communicate via internal IPs.

This causes problems when connecting to the master directly over other peered networks or VPN connections because the network peering routes to the master are not propagated over VPNs and network peering.

For your use case, disable the external master endpoint. Once this is done, when you run the get-credentials command, your kube config will have the internal master endpoint instead of the external one. You will then need to connect to your master (kubectl) from the VPC network (a bastion host or a proxy).

Instead, I recommend leaving the external end point active, use get-credentials without using --internal-ip so that your kube config will use the external end point and thus you can connect from anywhere. To make sure that you master is secure, use Master Authorized Networks to define the external IPs or CIDRs you will be connecting from

I am fairly certain the kubectl exec and logs commands are failing because of how you are getting the credentials.

One last thing worth checking as well, GKE creates firewall rules and routes automatically (they will be called gke-...), these are required to ensure that SSH tunnels from the master to the nodes work properly.