Kubernetes Nginx Ingress Controller – Metrics with Prometheus

ingresskubernetesmetricsnginxprometheus

I've tried to find a documentation about the metrics exposed by the NGINX ingress controller in Kubernetes but so far I haven't found any reliable source about the metrics and what they mean.

For example, there are three different request_size metrics (sum, bucket, count).

From my guessing sum and count correlate with each other. And to get the average request size I could probably do something like sum by (method) (request_size_sum{...}) / sum by (method) (request_size_count{...}).

But what about bucket and especially what is le?

Best Answer

For metrics itself and some explanations about them, I think the closest and full list is presented on Github issue - Document prometheus metrics

Note! This is about ingress nginx which is kubernetes community driven. Ingress nginx which is developed by nginx inc is a different project so there can be some differences.

As for the types of metrics, this is about Prometheus itself. There are different types of them:

counter:

A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors.
gauge:

A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.

Gauges are typically used for measured values like temperatures or current memory usage, but also "counts" that can go up and down, like the number of concurrent requests.
histogram (it's about buckets):

A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
summary:

Similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

Please also get familiar with histograms and summaries.

As for le:

le is the canonical abbreviation for "less than or equal".

Find an example of Prometheus queries and explanations

Related Solutions

Nginx-ingress controller crashes every few days

Kubernetes is a self healing, high availability and load balanced environment. Even if you have one node you still use cluster solutions to achieve best performance. In this kind of situation all software components are running on one machine. So there is a lot of software running - your application images and Kubernetes processes. This requires to have enough resources for every aspect of this configuration.

What if your application (that means docker image for example) is running out of resources? Kubernetes will kill it. Exhausted memory limits will cause OOO behavior to run and kill the container.

In log you've provided I found the following entry:

I0518 12:31:55.369911 7 main.go:150] Received SIGTERM, shutting down

So Kubernetes decided to kill Ingress controller due to exhausted resources. Try to extend the memory of the machine (or vm) and observe if it helps.

Another option is to start another node and extend cluster from one node to multi node with all features like HA and load balancing enabled to avoid issue like this.

Kubernetes nginx ingress session affinity

I found the answer, and whaddyaknow! It was my own fault, of course.

The problem was that I tried to configure the nginx ingress chart, which was not the way to go. The nginx ingress only provides the functionality - the services you run yourself provide the requested behaviour.

In my own service that I had deployed, I had an ingress-statement. Adding the annotation for cookies there solved my issue.

Also, creds to Nepomucen for informing me in the comments that the cookie affinity would work even though it's a bot user.

Here's my full helm chart for my own service that relied on the nginx ingress:

# Default values for chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
image:
  repository: [redacted]
  tag: [redacted]
  pullPolicy: IfNotPresent
  imagePullSecret: [redacted] # Must be registered with the namespace.
service:
  name: api
  type: ClusterIP
  externalPort: 8080
  internalPort: 8080
  livenessProbe: /alive
  readinessProbe: /ready
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: "cookie" # THIS WAS THE REQUIRED LINE.
    #kubernetes.io/tls-acme: 'true' # Kube Lego to obtain Let's Encrypt SSL certificates
  hosts:
    - [redacted]
  tls:
    - secretName: tls
      hosts:
        - [redacted]
hpa:
  minReplicas: 3
  maxReplicas: 10
  cpuAvg: 65
resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi
env:
- name: APPLICATION_ENV
  value: "production"
- name: TZ
  value: "Europe/Stockholm"

Best Answer

Related Solutions

Nginx-ingress controller crashes every few days

Kubernetes nginx ingress session affinity

Related Topic