Kubernetes v1.24.3 Upgrade Broke Ingress-Nginx – How to Fix

azurehelmkubernetesnginx-ingress

After upgrading our AKS kubernetes cluster to from v1.23.8 to v1.24.3 our ingress stopped working properly. No errors logged in events and the ingress-nginx pod does not report any errors on the console. Everything looks fine from within the cluster, but all ports for the public IP is closed externally.

Even curl'ing the web-apps that run in the cluster from within the cluster works fine.
It seems like it's just the opening of the ports externally that's broken.
Ingress-nginx is deployed via helm release (HR v4.2.5).

I have a feeling it must be some config for the ingress or controller that needs to be changed.

UPDATE: we did a new install of a plain AKS cluster and did helm install quickstart ingress-nginx/ingress-nginx in 1.23.8 (which works), 1.24.0 (which does not work) and in 1.24.3 (which does not work either).

Any ideas or pointers?

Best Answer

We found the issue.

For clusters v1.24.0 and up the health probes for the load balancer is set to HTTP and HTTPS instead of TCP. When we changed the health probes to use TCP it all worked again.

Created an issue for AKS on this: https://github.com/Azure/AKS/issues/3210

The proper fix was to add the following annotation to the nginx service (see link to AKS issue above):

values:
controller:
  service: 
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

Related Solutions

Kubernetes nginx ingress session affinity

I found the answer, and whaddyaknow! It was my own fault, of course.

The problem was that I tried to configure the nginx ingress chart, which was not the way to go. The nginx ingress only provides the functionality - the services you run yourself provide the requested behaviour.

In my own service that I had deployed, I had an ingress-statement. Adding the annotation for cookies there solved my issue.

Also, creds to Nepomucen for informing me in the comments that the cookie affinity would work even though it's a bot user.

Here's my full helm chart for my own service that relied on the nginx ingress:

# Default values for chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
image:
  repository: [redacted]
  tag: [redacted]
  pullPolicy: IfNotPresent
  imagePullSecret: [redacted] # Must be registered with the namespace.
service:
  name: api
  type: ClusterIP
  externalPort: 8080
  internalPort: 8080
  livenessProbe: /alive
  readinessProbe: /ready
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: "cookie" # THIS WAS THE REQUIRED LINE.
    #kubernetes.io/tls-acme: 'true' # Kube Lego to obtain Let's Encrypt SSL certificates
  hosts:
    - [redacted]
  tls:
    - secretName: tls
      hosts:
        - [redacted]
hpa:
  minReplicas: 3
  maxReplicas: 10
  cpuAvg: 65
resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi
env:
- name: APPLICATION_ENV
  value: "production"
- name: TZ
  value: "Europe/Stockholm"

K8s nginx ingress returns randomly 502 error on load

if you use the nginx ingress reverse proxy, it maybe the cause. Try to check in Configmap for proxy-next-upstream settings, and extend it to handle the http_502 case.
Also enable the retry-non-idempotent if the request that got 502 is POST, LOCK, PATCH, if it is safe for your app to do so.

My guess: when the backend pod reach its load limit (or pod recycling), it "rejects" new request and nginx reverse proxy is more sensitive with this rejection. Without nginx-ingress, k8s handling the load balancing and it can handle everything better, queuing instead of rejecting.

Best Answer

Related Solutions

Kubernetes nginx ingress session affinity

K8s nginx ingress returns randomly 502 error on load

Related Topic