Kubernetes Service – How to Route Traffic to Different Pods Based on Port

kubernetes

I would like to have a Service of LoadBalancer type that points to an nginx on port 80, and point to a separate sshd pod for port 22. However I can't route based on the port with a single selector.

The use case I'm going for is similar to github.com which accepts traffic to github.com for port 80 as well as 22 for ssh traffic. So DNS would point to a set of k8s loadbalancers in this case that I would assume would route to the appropriate pods per port. So if I'm going at it the wrong way, lemme know. I'm open to other solutions.

What I would like to avoid doing is setting up a separate pod like HAProxy that routes per port.

I looked into using an Ingress, but that is only for HTTP traffic.

An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.

https://kubernetes.io/docs/concepts/services-networking/ingress/

Best Answer

What I would like to avoid doing is setting up a separate pod like HAProxy that routes per port.

You don't need to set up such separate Pod.

The Kubernetes Ingress by default does not support TCP or UDP services. But for example, ingress-nginx controller provides a mechanism to support TCP or UDP on different ports. You can expose TCP or UDP ports by modifying ConfigMaps.

For this reason, this Ingress controller uses the flags '--tcp-services-configmap' and '--udp-services-configmap' to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format:

<namespace/service name>:<service port>:[PROXY]:[PROXY]

Check additional info here.

Such ConfigMap should already be available before deploying the Ingress Controller.

So, try to:

1. Create a ConfigMap with the following TCP service configuration.

$ cat ingress-nginx-tcp.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-tcp
  namespace: default  
data:
  "22": targetnamespace/target-service:22

2. Point Ingress controller to this ConfigMap using the --tcp-services-configmap flag in the configuration like this:

$ kubectl get deployment ingress-nginx-controller -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: default
spec:
...
  template:
...
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp
...

3. Expose port 22 in the Service defined for the Ingress like this:

$ kubectl get svc ingress-nginx-controller -o yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: default
spec:
  ports:
  - name: tcp-22
    nodePort: 30957
    port: 22
    protocol: TCP
    targetPort: 22
...
  type: LoadBalancer
...

You can define any number of ports that can be exposed using this method.

There is another option for those who are using an ingress-nginx helm chart. Most of the configuration is already done, and you just need to specify your ports in tcp section like this:
tcp: 
  2222: "default/example-tcp-svc:22"

where 2222 is the exposed port and 22 is the service port.

Related Topic