I don't know where to look for hints.
We have installed gitlab-runners using a helm chart in our development cluster. Most of the time this works, but in the last week or so we have experienced pods being stuck in Pending
state without any further logs. At some point which I cannot define better, all pods are being scheduled on nodes, then the next batch is stuck in Pending
again.
We use GKE and have set up a node pool of preemtible nodes only for gitlab-runner pods. We run kubernetes v1.15.4-gke.18
.
We know there are several reasons for pods being stuck in Pending
but I always expect some form of logs/indication when running kubectl describe <GITLAB_RUNNER_POD>
or kubectl get events
. The problem is, there is none. No events.
We have stackdriver logging enabled and I can see Kubernetes Apiservice Requests
logs under Kubernetes Cluster
but they don't have any meaningful content to me.
Any ideas where to look?
Best Answer
Posting this answer to give more of a general idea for where to look for information why
Pod
is inPending
state as for now it's impossible to tell on this specific setup.The ways to check why the
Pod
can be inPending
state:$ kubectl describe pod POD_NAME
$ kubectl get events -A
Cloud Logging
(more on that below)Assuming following situation where the
Pod
is inPending
state:$ kubectl get pods
To get more information about it's state you can run:
$ kubectl describe pod POD_NAME
The
Event
part of above output:As you can see there is an information on why the
Pod
is inPending
state (Insufficient CPU
).You can also run:
$ kubectl get events
Retrieving logs from
Cloud Logging
:You can run below query to get the
Pods
that were inPending
state:This query will not show the reason (like
Insufficient CPU
) of whyPod
is inPending
state. There is a feature request on Issuetracker.google.com for this reason. You can follow it to receive further updates:Additional resources: