Nginx – How to properly configure access to kubernees dashboard behind nginx ingress

kubernetesnginxnginx-ingress

I'm trying to configure nginx ingress to access several services, like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-monit
spec:
  rules:
  - host: grafana.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: prometheus-grafana
          servicePort: 80
  - host: kubernetes-dashboard.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

I've access to the grafana service without any problems, my issue is with kubernetes-dashboard.
I've already configured kubernetes-dashboard to allow HTTP traffic with this configuration

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: monit
spec:
  ports:
    - port: 80
      targetPort: 9090
  selector:
    k8s-app: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: monit
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.0.0-beta8
          imagePullPolicy: Always
          ports:
            - containerPort: 9090
              protocol: TCP
          args:
            - --namespace=monit
            - --insecure-bind-address=0.0.0.0
            - --insecure-port=9090
            - --enable-insecure-login
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 9090
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "beta.kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

I;ve also a valid token which I can use to access kubernetes dashboard when I use ClusterIP.
However when I access it through ngress I cannot go over the login page even with valid token (see screenshot).

enter image description here

I looked into Nginx logs for problems/errors but everything seemed fine

$ kubectl logs -n monit ingress-nginx-controller-bbdc786b4-6nl9h  -f
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "GET /api/v1/csrftoken/login HTTP/1.1" 200 85 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 479 0.001 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 85 0.001 200 59fc952888dfadf0223740c31e562ef8
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "POST /api/v1/login HTTP/1.1" 200 1508 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 1545 0.005 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 1508 0.005 200 241388246b11031765557475bea603ff
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "GET /api/v1/plugin/config HTTP/1.1" 200 185 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 477 0.003 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 185 0.003 200 45371469793ce4f35c45dec70530bea0
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "GET /api/v1/login/status HTTP/1.1" 200 108 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 476 0.001 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 108 0.001 200 49171f5e9316a2d6da883d1c4f0b50df
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "GET /api/v1/login/status HTTP/1.1" 200 108 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 476 0.001 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 108 0.001 200 c69b9d166f1527f00e7cd175696ec8c7
192.168.65.3 - - [03/Jun/2020:02:03:13 +0000] "GET /api/v1/login/status HTTP/1.1" 200 108 "http://kubernetes-dashboard.localhost/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" 476 0.001 [monit-kubernetes-dashboard-80] [] 10.1.0.123:9090 108 0.001 200 1f9c27ca407bca57dcc0c26bca65be58

What am I missing in my ingress configuration?

UPDATE: I tried to setup an https ingress for the dashboard with this config

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: https-ingress-monit
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: kubernetes-dashboard.localhost
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 443

But this does not seem to work, no endpoint is configured

$ kubectl describe ingress https-ingress-monit -n monit
Name:             https-ingress-monit
Namespace:        monit
Address:          localhost
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  kubernetes-dashboard.localhost  
                                  /   kubernetes-dashboard:443 (<error: endpoints "kubernetes-dashboard" not found>)
Annotations:                      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  87s   nginx-ingress-controller  Ingress monit/https-ingress-monit
  Normal  UPDATE  74s   nginx-ingress-controller  Ingress monit/https-ingress-monit

Now when I try to access http://kubernetes-dashboard.localhost/ I see 503 Service Temporarily Unavailable

Best Answer

TL;DR

You cannot pass the verification (pressing Sign In doing nothing) by the lack of HTTPS.

As I said in the comment:

Login not available

If your login view displays below error, this means that you are trying to log in over HTTP and it has been disabled for the security reasons.

Logging in is available only if URL used to access Dashboard starts with:

  • http://localhost/...
  • http://127.0.0.1/...
  • https://<domain_name>/...

Github.com: Kubernetes: Dashboard: Login not available

You can login to Kubernetes Dashboard without HTTPS only with:

  • http://localhost/...
  • http://127.0.0.1/...

You need HTTPS to login to your Kubernetes Dashboard with:

  • https://IP.ADDRESS
  • https://DOMAIN.NAME

endpoints "kubernetes-dashboard" not found

But this does not seem to work, no endpoint is configured

It means that the Ingress resource could not find the Endpoint to sent the traffic to. This happened in your case because:

  • Ingress is in default namespace
  • Service named kubernetes-dashboard is in monit namespace

To make it work you can (one of the ways) make another Ingress resource specifically in monit namespace.

You can invoke below commands to get more information about your resources:

  • $ kubectl get services -n monit
  • $ kubectl get endpoints -n monit

Resources in Kubernetes are strictly connected to namespaces. You can read about them more here: Kubernetes.io: Concepts: Working with objects: Namespaces

You have several ways to deploy Kubernetes Dashboard. It will depend on the solution you are using (minikube,bare metal kubeadm cluster,eks,gke, etc.).

General steps to take to deploy Kubernetes Dashboard with Nginx-ingress:

  • Deploy Nginx-ingress
  • Download and modify the Dashboard definition
  • Configure access to Dashboard with Ingress
  • Test it

Deploy Nginx-ingress

Please follow official documentation regarding deployment of Nginx-ingress: Kubernetes.github.io: Ingress-nginx: Deploy

Download and modify the Dashboard definition

Installation of Kubernetes Dashboard: Kubernetes.io: Web ui dashboard: Deployment

Above link can be used to deploy Dashboard but few adjustments will need to be made.

Assume the following:

  • Every resource in kubernetes-dashboard namespace
  • Arguments for Dashboard: 
      - args:
    - --namespace=kubernetes-dashboard
    - --enable-insecure-login
    - --insecure-bind-address=0.0.0.0
  • Dashboard listening on port 9090
  • Services as well as healthchecks associated with Dashboard set to port 9090/TCP/HTTP.

A tip for arguments!

enable-skip-login false When enabled, the skip button on the login page will be shown.

Github.com: Kubernetes: Dashboard: Arguments

Your Dashboard definition will need a Service to be exposed outside the cluster. You can create your own definition of a Service like example below or edit definition included in installation YAML above.

Example below: 

kind: Service
apiVersion: v1
metadata:
  name: dashboard-service
  namespace: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
    - port: 80
      targetPort: 9090
      nodePort: 30001
      name: dashboard-port
  type: NodePort

Please take a specific look on part: 

  ports:
    - port: 80
      targetPort: 9090
      nodePort: 30001
      name: dashboard-port

Traffic will be sent to the Dashboard pod on port 9090 as required by the arguments of the Dashboard itself.

Configure access to Dashboard with Ingress

Assuming your Ingress is deployed correctly you can use below example to expose Dashboard:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - secretName: tls-secret # NON-EXISTENT
  rules:
  - host:
    http:
      paths:
      - path: /
        backend:
          serviceName: dashboard-service
          servicePort: dashboard-port

Please take specific look on parts:

  • - secretName: tls-secret # NON-EXISTENT - it will configure controller to use a fake certificate and allow HTTPS connections
  • namespace: kubernetes-dashboard - namespace is exactly the same as other Dashboard resources
  • serviceName: dashboard-service - name of the service associated with Dashboard
  • servicePort: dashboard-port - name of the port of the service associated with Dashboard

Test it

After this steps you should be able to enter either ip address or domain name to your web browser and open Dashboard panel.

Please make sure that you connect to Dashboard with: https://.

If you configured your Dashboard to require authentication you should provide the authentication token. You can find your token by invoking below command:

  • $ kubectl describe secret NAME_OF_THE_SECRET -n NAMESPACE
Related Topic