I have an ingress that routes to a custom endpoint external to the kubernetes cluster. The service listens only on HTTPS on port 8006.
apiVersion: v1
kind: Service
metadata:
name: pve
spec:
ports:
- protocol: TCP
port: 8006
---
apiVersion: v1
kind: Endpoints
metadata:
name: pve
subsets:
- addresses:
- ip: 10.0.1.2
ports:
- port: 8006
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: pve
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/auth-tls-verify-client: "off"
nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/16"
spec:
tls:
- hosts:
- pve.example.com
secretName: pve-tls
rules:
- host: pve.example.com
http:
paths:
- backend:
serviceName: pve
servicePort: 8006
path: /
Gives the error in the nginx pod:
10.0.0.25 – – [28/Aug/2020:01:17:58 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0" "-"
2020/08/28 01:17:58 [error] 2609#2609: *569 upstream prematurely closed connection while reading response header from upstream, client: 10.0.0.25, server: pve.example.com, request: "GET / HTTP/1.1", upstream: "http://10.0.1.2:8006/", host: "pve.example.com"
Edit
After removing the proxy protocol, I get the error
10.0.10.1 – – [28/Aug/2020:02:19:18 +0000] "GET / HTTP/1.1" 400 59 "-" "curl/7.58.0" "-"
2020/08/28 02:19:26 [error] 2504#2504: *521 upstream prematurely closed connection while reading response header from upstream, client: 10.0.10.1, server: pve.example.com, request: "GET / HTTP/1.1", upstream: "http://10.0.1.2:8006/", host: "pve.example.com"
10.0.10.1 – – [28/Aug/2020:02:19:26 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.58.0" "-"
And in case it is relevant, my nginx configuration, deployed through the helm char nginx-stable/nginx-ingress
## nginx configuration
## Ref: https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md
##
controller:
config:
entries:
hsts-include-subdomains: "false"
ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
ssl-protocols: "TLSv1.1 TLSv1.2"
ingressClass: nginx
service:
externalTrafficPolicy: Local
annotations:
metallb.universe.tf/address-pool: default
defaultBackend:
enabled: true
tcp:
22: "gitlab/gitlab-gitlab-shell:22"
Best Answer
This annotation is probably the cause of the problem.
The docs state:
If you don't have a load balancer in front of your Ingress which is passing connections in using the PROXY protocol, then this is not what you want, and this annotation should not be present (or should be
"false"
).