I've recently upgraded my on-premises Kubernetes cluster from 1.12.1 to 1.13.0 for obvious reasons, but now my kubectl running on the master node cannot get the container logs anymore from any node, except the master itself.
This issue happens on both the nodes where kubelet has been upgraded to 1.13.0, but also on the nodes where kubelet is still running under version 1.12.1. This makes me think that it is not the problem with kubelet itself, but something in the configuration handling in version 1.13.0 changed that causes the logging to fail.
When I do a kubectl logs -f pod/podname-749c5c589f-vt7hj
, I get:
Error from server: Get https://10.48.76.46:10250/containerLogs/default/podname-749c5c589f-vt7hj/podname?follow=true: Forbidden
I've checked all the obvious things like network connections between the master and the minion running this pod, as any proxies that might be configured, but it keeps failing. Also the kubelet log on the minion node doesn't log any error.
Furthermore, other usual kubelet commands for this pod, like attach
and exec
work fine for this pod, so the inter-node communication is working fine. Also the pod itself is running fine, I just cannot see its logs.
Any idea what happened in 1.13.0 that might cause this to break?
Best Answer
I had a no_proxy environment variable for the hostnames of the minion nodes in the kube-apiserver.yaml configuration file, but for some reason the kube-apiserver connected to the IP addresses of the minion nodes and not to the hostnames. It did not do this before in version 1.12.1, as it only connected to the master node via its IP address. I have added the IP addresses of all the minion nodes in the config, restarted kubelet (and thus kube-apiserver) and the issue is now solved.
Protip: If you need to use a http/https proxy to pull images from the internet. make sure you add both the hostname and IP address of all master and minion nodes in your no_proxy environment variable.