Background
I think your issue is cased by your 2 Flannel CNI
pods CrashLoopBackOff
status.
Your error
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8eee497a2176c7f5782222f804cc63a4abac7f4a2fc7813016793857ae1b1dff" network for pod "demo-6c59fb8f77-9x6sr": networkPlugin cni failed to set up pod "demo-6c59fb8f77-9x6sr_default" network: open /run/flannel/subnet.env: no such file or directory
is pointing that pod cannot be created due to lack of /run/flannel/subnet.env
file.
In Flannel Github document you can find:
Flannel runs a small, single binary agent called flanneld on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space.
Meaning, to proper work, Flannel
pod should be running on each node as it contains subnets information. From your outputs I can see that only 1 is working properly out of 3 Flannel
pods.
NAMESPACE NAME READY STATUS RESTARTS AGE
...
kube-system kube-flannel-ds-kqz4t 0/1 CrashLoopBackOff 92 27h
kube-system kube-flannel-ds-szqzn 1/1 Running 3 27h
kube-system kube-flannel-ds-v9q47 0/1 CrashLoopBackOff 142 27h
If mentioned pod was scheduled on node where flannel pod is not working it won't be created due to CNI network issues
. Besides your demo
pod, also kubernetes-dashboard
pods have the same issue with ContainerCreating
status.
Conclusion
Your demo
pod cannot be scheduled as Kubernetes encounter some network issues related with flannel configuration file (...network: open /run/flannel/subnet.env: no such file or directory
).
Your flannel pods restarts counts is very high as for 27 hours
. You have to determine why and fix it. It might be lack of resources, network issues with your infrastructure or many other reasons. Once all flannel
pods will be working correctly, your shouldn't encounter this error.
Solution
You have to make flannel pods works correctly on each node.
Additional Troubleshooting Details
For detailed investigation please provide
$ kubectl describe kube-flannel-ds-kqz4t -n kube-system
$ kubectl describe kube-flannel-ds-v9q47 -n kube-system
Logs details would be also helpful
$ kubectl logs kube-flannel-ds-kqz4t -n kube-system
$ kubectl logs kube-flannel-ds-v9q47 -n kube-system
Please replace kubectl get pods --all-namespaces
with kubectl get pods -o wide -A
and output of kubectl get nodes -o wide
.
If you will provide those information, it should be possible to determine root cause of flannel
pods issues and I will edit this answer with exact solution.
Best Answer
Found a way to access remote kubernetes cluster without ssh'ing to one of the nodes in cluster. You need to edit ~/.kube/config file as below :
Then set context by executing:
After this you should be able to interact with the cluster.
Note : To add certification and key use following link : http://kubernetes.io/docs/user-guide/kubeconfig-file/
Alternately, you can also try following command