I'm following guide from Linux Foundation "Kubernetes Administrator" course and stuck on deploying simple app. I think trouble is even earlier than with app deployment.
I've created master and worker, seems that they are ok:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-training-server-1 Ready master 63m v1.19.1
ubuntu-training-server-2 Ready <none> 57m v1.19.1
But here is something wrong:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-6799fc88d8-556z4 0/1 ContainerCreating 0 50m
kube-system calico-kube-controllers-69496d8b75-thcl8 1/1 Running 1 63m
kube-system calico-node-gl885 0/1 CrashLoopBackOff 20 58m
kube-system calico-node-jvc59 1/1 Running 1 63m
kube-system coredns-f9fd979d6-hjfst 1/1 Running 1 64m
kube-system coredns-f9fd979d6-kvx42 1/1 Running 1 64m
kube-system etcd-ubuntu-training-server-1 1/1 Running 1 64m
kube-system kube-apiserver-ubuntu-training-server-1 1/1 Running 1 64m
kube-system kube-controller-manager-ubuntu-training-server-1 1/1 Running 1 64m
kube-system kube-proxy-9899t 1/1 Running 1 58m
kube-system kube-proxy-z6b22 1/1 Running 1 64m
kube-system kube-scheduler-ubuntu-training-server-1 1/1 Running 1 64m
I mean not all are ready.
If I try to get details about trouble node, I see:
$ kubectl logs -n kube-system calico-node-gl885
Error from server (NotFound): the server could not find the requested resource ( pods/log calico-node-gl885)
And when I try to deploy nginx I get:
$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
and
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/1 1 0 52m
Here are troubles too:
$ kubectl get events
...
92s Warning FailedCreatePodSandBox pod/nginx-6799fc88d8-556z4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4c451bc2c92f555c930f84e4e8b7082a03dd2824cf50948d348893ebea488d93" network for pod "nginx-6799fc88d8-556z4": networkPlugin cni failed to set up pod "nginx-6799fc88d8-556z4_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
...
I see no /var/lib/calico/nodename
on worker node, only on master and in guide there was speaking only about kubectl apply -f calico.yaml
on master.
Could anybody help me get rid of calico errors? Tried to search, have seen similar cases but looks like that they are about something different.
UPDATE
I've found possible networking conflict (Calico config contained 192.168.0.0/16
and my VirtualBox adapter was 192.168.56.0/24
) so I reset cluster, changed Calico config and networking/podSubnet
in kubeadm-config.yaml
to 192.168.0.0/24
and init cluster again.
New status is following.
Seem OK:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-training-server-1 Ready master 39m v1.19.1
ubuntu-training-server-2 Ready <none> 38m v1.19.1
Seem OK too:
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
36m Normal Starting node/ubuntu-training-server-1 Starting kubelet.
36m Normal NodeHasSufficientMemory node/ubuntu-training-server-1 Node ubuntu-training-server-1 status is now: NodeHasSufficientMemory
36m Normal NodeHasNoDiskPressure node/ubuntu-training-server-1 Node ubuntu-training-server-1 status is now: NodeHasNoDiskPressure
36m Normal NodeHasSufficientPID node/ubuntu-training-server-1 Node ubuntu-training-server-1 status is now: NodeHasSufficientPID
36m Normal NodeAllocatableEnforced node/ubuntu-training-server-1 Updated Node Allocatable limit across pods
36m Normal NodeReady node/ubuntu-training-server-1 Node ubuntu-training-server-1 status is now: NodeReady
35m Normal RegisteredNode node/ubuntu-training-server-1 Node ubuntu-training-server-1 event: Registered Node ubuntu-training-server-1 in Controller
35m Normal Starting node/ubuntu-training-server-1 Starting kube-proxy.
35m Normal Starting node/ubuntu-training-server-2 Starting kubelet.
35m Normal NodeHasSufficientMemory node/ubuntu-training-server-2 Node ubuntu-training-server-2 status is now: NodeHasSufficientMemory
35m Normal NodeHasNoDiskPressure node/ubuntu-training-server-2 Node ubuntu-training-server-2 status is now: NodeHasNoDiskPressure
35m Normal NodeHasSufficientPID node/ubuntu-training-server-2 Node ubuntu-training-server-2 status is now: NodeHasSufficientPID
35m Normal NodeAllocatableEnforced node/ubuntu-training-server-2 Updated Node Allocatable limit across pods
22s Normal CIDRNotAvailable node/ubuntu-training-server-2 Node ubuntu-training-server-2 status is now: CIDRNotAvailable
35m Normal Starting node/ubuntu-training-server-2 Starting kube-proxy.
35m Normal RegisteredNode node/ubuntu-training-server-2 Node ubuntu-training-server-2 event: Registered Node ubuntu-training-server-2 in Controller
35m Normal NodeReady node/ubuntu-training-server-2 Node ubuntu-training-server-2 status is now: NodeReady
And here is new trouble, calico-kube-controllers-69496d8b75-gdbd7
is starting for more than half of an hour:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-69496d8b75-gdbd7 0/1 ContainerCreating 0 37m
kube-system calico-node-8xjsm 0/1 CrashLoopBackOff 13 37m
kube-system calico-node-zktsh 1/1 Running 0 37m
kube-system coredns-f9fd979d6-7bkwn 1/1 Running 0 39m
kube-system coredns-f9fd979d6-rsws5 1/1 Running 0 39m
kube-system etcd-ubuntu-training-server-1 1/1 Running 0 39m
kube-system kube-apiserver-ubuntu-training-server-1 1/1 Running 0 39m
kube-system kube-controller-manager-ubuntu-training-server-1 1/1 Running 0 39m
kube-system kube-proxy-2tvjp 1/1 Running 0 39m
kube-system kube-proxy-jkzbz 1/1 Running 0 39m
kube-system kube-scheduler-ubuntu-training-server-1 1/1 Running 0 39m
UPDATE 2
Details about my setup.
$ cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.19.1
controlPlaneEndpoint: "k8smaster:6443"
networking:
podSubnet: 192.168.0.0/24
Cluster was initialized with:
kubeadm init --config=kubeadm-config.yaml --upload-cert
Best Answer
Got it. In my setup I've got two VirtualBox VMs both with two network interfaces - one to connect with outer world (10.0.2.15) and one to talk to each other (192.168.56.104, 192.168.56.105). In
kubeadm init
log I've found that it was using first one, so I've explicitly toldkubeadm
to use internal IP. Here is command with which I've succeeded to create cluster and deploy simple app into itOne sad thing - unfortunately I could not find how to add to config options which I've used in command line.