Kubernetes Nginx Ingress – Fixing HTTP-01 Challenge Propagation Issues

I'm having issues with my rapberry pi kubernetes implementation

Problem:

I have cert-manager letsencrypt ACME challenge waiting due to a 401 error code on bare metal kubernetes install.

Setup

Platform: Raspberry Pi 4

OS: Ubuntu Server 20.04.3 LTS 64 bit

Ingress: Nginx

Loadbalancer: Metallb

Networking: Calico

I installed metallb and nginx via helm using:

helm install metallb metallb/metallb --namespace kube-system\
    --set configInline.address-pools[0].name=default\
    --set configInline.address-pools[0].protocol=layer2\
    --set configInline.address-pools[0].addresses[0]=<ip-range>

and

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace kube-system

My letsencrypt looks like this:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: <email redacted>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

My nginx ingress set up looks like this:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: "nextcloud" # Same namespace as the deployment
  name: "nextcloud-ingress" # Name of the ingress (see kubectl get ingress -A)
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # Encrypt using the ClusterIssuer deployed while setting up Cert-Manager
    nginx.ingress.kubernetes.io/proxy-body-size:  "125m" # Increase the size of the maximum allowed size of the client request body
spec:
  tls:
  - hosts:
    - "nextcloud.<domain redacted>" # Host to access nextcloud
    secretName: "nextcloud-prod-tls" # Name of the certificate (see kubectl get certificate -A)
  rules:
  - host: "nextcloud.<domain redacted>" # Host to access nextcloud
    http:
      paths:
        - path: /  # We will access NextCloud via the URL https://nextcloud.<domain.com>/
          pathType: Prefix
          backend:
            service: 
              name: "nextcloud-server" # Mapping to the service (see kubectl get services -n nextcloud)
              port: 
                number: 80 # Mapping to the port (see kubectl get services -n nextcloud)
---

Debugging

When I look at the ingress controller logs (different namespace) I see:

Service "nextcloud/cm-acme-http-solver-9tccf" does not have any active Endpoint.

But the endpoint appears to exist when I do kubectl get endpoints -A

My certificate exists as:

kubectl get certificate -n nextcloud
NAME                 READY   SECRET               AGE
nextcloud-prod-tls   False   nextcloud-prod-tls   3h58m

Following the recommended debug steps from cert manager I tracked the issue to the challenges whereby I get:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '401', expected '200'
  State:       pending
Events:        <none>

I'm kinda stuck I've been googling my heart out but there doesn't seem to be a lot on this. I'm guessing I've stuffed up on the set up but I've mainly been following the documentation on the relevant pages. Any pointers would be greatly appreciated :). If you need any additional info let me know this is currently quite long so I tried to include what I thought were problem points.