Can a Kubernetes pod span nodes

kubernetes

A pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), the shared storage for those containers, and options about how to run the containers. Pods are always co-located and co-scheduled, and run in a shared context. A pod models an application-specific “logical host” – it contains one or more application containers which are relatively tightly coupled — in a pre-container world, they would have executed on the same physical or virtual machine.

But I'm unsure if "co-location" means on the same P̶o̶d̶ Node. When satellites are co-located they are "in close proximity so that to reception equipment on the ground they 'appear' to occupy a single orbital position." So, that could be on the same Cluster, not on the same Node.

At the time of posting this, the only info I can find on this issue is at https://platform9.com/blog/compare-kubernetes-vs-ecs/ where it says…

Containers in a single Pod are guaranteed to run on a single Kubernetes node.

Now that sounds pretty conclusive, but I haven't been able to get any corroboration on that anywhere. I want to make sure that they are correct and didn't just misunderstand and spread misinformation.

The reason I'm asking is that it seems like Kubernetes is inherently wasteful if this is true. It is almost certain that your nodes with pods deployed will have resources left over. It is likely that you will have a pod whose needs could be satisfied by the combination of these left over resources. However, if a pod cannot span multiple nodes, you will have to create a new node and have even more un-utilized resources.

Best Answer

The very idea of pods is that of co-location, that is, being able to define compute-locality where it makes sense (or where it's needed). So, no, per definition, since all containers in a pod will be scheduled on the same node, a pod can not span nodes.

See also my critique of pods here. Having said that, we've introduced the pods concept in DC/OS now with v1.9 as well. There seems to be a certain demand, but my hunch (since I don't have data on it) is that it's mostly a sort of safety blanket for cases where one doesn't or can't go all in, cloud-native wise.

Related Solutions

Networking – How to Access Kubernetes NodePort from Other Worker Nodes

I've found the problem at last.

It's a combination of flannel, complicated cni network structure and virtualization.

flannel is implementing its node to node routing network with vxlan and it's enabling offloading features on the vxlan device.

When offloading is enabled on a flannel device, NodePort accesses across nodes experiences a 63 second routing delay. The delay goes away when offloading is turned off with

ethtool --offloading tx off rx off flannel.1

My cluster is working on a Proxmox environment and the network cards are virtio devices. I don't think the problem is on the virtualization layer because I have another cluster on the same Proxmox node and everything is working fine. Since it's a production cluster, it's not updated this frequently.

So this one is marked as a flannel bug. Now I'll try to update flannel and see what happens.

Update: flannel is updated on this particular cluster and it's working like charm.

Docker – Kubernetes Pod OOMKilled Issue

First of all, the pod should not request more memory/CPU just because the node has gotten more resources. Without your specs it is hard to point out what might be wrong config wise but I would like to explain the concept to make it more clear.

You mentioned your pod configuration but did not specify if these are limits or requests.

Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. These will not cause OOMs, they will cause pod not to get scheduled.
Limits, on the other hand, make sure a container never goes above a certain value. This can cause OOM kill.

Requests and limits are on a per-container basis. While Pods usually contain a single container, it’s common to see Pods with multiple containers as well. Each container in the Pod gets its own individual limit and request, but because Pods are always scheduled as a group, you need to add the limits and requests for each container together to get an aggregate value for the Pod.

Below is an example of a pod that has 2 containers with the same specified requests and limits:

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: wp
    image: wordpress
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

If you want to calculate requests and limits for the whole pod you need to sum those values, making it: request of 0.5 cpu and 128 MiB of memory, and a limit of 1 cpu and 256MiB of memory.

If you want to find more regarding that topic than check out the official documentation:

Please, let me now if that helped.

Best Answer

Related Solutions

Networking – How to Access Kubernetes NodePort from Other Worker Nodes

Docker – Kubernetes Pod OOMKilled Issue

Related Topic