When adding a new node to a Kubernetes cluster I end up with this error :
+ docker start kubelet
Error response from daemon: {"message":"No such container: kubelet"}
Error: failed to start containers: kubelet
+ sleep 2
This error occurs on a cluster that is already damaged. There is only one node left out of 3. The node remaining a priori a problem of certificate recovery and distribution. SSL is no longer functional on this node. For information, the Kubernetes cluster has been deployed through Rancher. The etcd container regularly reboots on node 3, and etcd does not want to deploy to the other nodes that I am trying to re-integrate into the cluster.
Kubelet is launched in a Docker container, itself launched by Rancher when he created the Kubernetes cluster. Regarding the tests carried out, I relaunched a new docker container with etcd, I tried to start again from a snapshot … nothing allows to relaunch the cluster. Adding a new node is also not functional. From what I've seen, there is also an issue with ssl certificates created by Rancher that he cannot find
Best Answer
Try to do following steps:
This will delete all the Docker volumes, be careful if you have
important data in your volumes.
The official docs on node cleanup recommend also removal of
/opt/rke
and/var/lib/etcd
. You cannot remove them because they contain cluster etcdsnapshots and data. This is especially important in case there's only one node
in the cluster.
exec
-ed into the rancher container and hacked the cluster status (thx@ibrokethecloud for the hint):
Inside the container:
The editor find the key
apiEndpoint
(it should be directly underthe
status
key) and removed it. Exit the editor and container. Make surekubectl says that it updated the cluster.
Set a different name for the node than it was before by adding a
--node-name
to the docker run command (actually there's an edit box for thisunder advanced settings). It looked like this:
successfully and RKE started up all the
kube-*
andkubelet
containers.Take a look: rancher-kubelet, rancher-2-getting-started.