Kubernetes CNI Hairpin Traffic

kubernetesnetworking

I setup a Kubernetes cluster using Kubernetes the Hard Way tutorial, and the connection is hanging whenever a Pod connects to another Pod on the same node through a ClusterIP (hairpin traffic).

If I access the pods directly, without going through the ClusterIP, everything works fine.

So, visually, this doesn't work:

PodA -> ServiceA ClusterIP -> PodA 
PodA -> ServiceB ClusterIP -> PodB on same node

However, this works 100% great, as does any Pod contacting another Pod directly by it's IP:

PodA -> ServiceB ClusterIP -> PodB on other node

I found Kubernetes Documentation about debugging services and went through it, and everything seems fine, up to the section A Pod can't reach itself via Service VIP.

I see rules added for my services in iptables-save output (and I confirmed using iptables mode):

-A KUBE-SERVICES ! -s 10.200.0.0/16 -d 10.32.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.32.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 10.200.0.0/16 -d 10.32.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.32.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4

I can see from the kubelet's logs promiscuous-bridge hairpin mode flag:

kubelet[12496]: I1204 04:13:29.761707   12496 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge"

I don't see logs such as Hairpin mode set to "promiscuous-bridge" specifically confirming the mode, so I set it explicitly in the kubelet-config.yml also

Also, I edited the CNI plugin to add promiscMode: true (docs), and I see PROMISC on the cnio0 interface:

cnio0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500

At this point I think either 1) this tutorial doesn't have hairpin traffic working or 2) I screwed up something obscure that's breaking this, but I can't figure out what it is!

Since this Kubernetes the Hard Way tutorial is known to be the canonical setup reference, I'm doubting it would be #1… anyone have other suggestions to determine #2?

Best Answer

This is a known issue with CNI documented here and here.

I think the abstract "allows hairpin behavior" is a property of plugins. Some support it. Some do not. Some may support it in multiple ways with configuration required. If you don't support it in a Kubernetes install, one class of Service behavior will not work correctly. That may be OK for some installations and not for others.

However, you can use Flannel with hairpinMode set to true (the default flannel configuration does not set HairpinMode to true). That pat of the config would look like this:

    {
      "name": "<name_here>",
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    }

Please let me know if that helped.

Related Topic