HAProxy – Resolving NOSRV Responses Despite Backend Being Up

Im making the same request(POST /user/oauth) to haproxy but sometimes i get 503, sending requests directly to the backend always works

here is the logs:

1. local_server~ app/s2 0/0/0/100/100 200 813 – – –VN 1/1/0/0/0 0/0
   "POST /user/oauth HTTP/1.1"
2. local_server~ app/s1 0/0/1/107/108 200 820 – – –NI 1/1/0/0/0 0/0
   "POST /user/oauth HTTP/1.1"
3. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 – – SCNN 1/1/0/0/0
   0/0 "POST /user/oauth HTTP/1.1"
4. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 – – SCNN 1/1/0/0/0
   0/0 "POST /user/oauth HTTP/1.1"
5. local_server~ app/s2 0/0/1/106/107 200 831 – – –NI 2/2/0/0/0 0/0
   "POST /user/oauth HTTP/1.1"
6. local_server~ app/s1 0/0/1/3/6206 101 337 – – –NI 3/3/1/1/0 0/0
   "GET / HTTP/1.1"
7. local_server~ app/s1 0/0/1/83/84 200 258 – – –VN 2/2/0/0/0 0/0
   "POST /transaction/concludeTransaction HTTP/1.1"
8. local_server~ app/NOSRV 0/-1/-1/-1/0 503 216 – – SCNN 1/1/0/0/0
   0/0 "POST /user/oauth HTTP/1.1"
9. local_server~ app/s2 0/0/1/107/108 200 831 – – –NI 1/1/0/0/0 0/0
   "POST /user/oauth HTTP/1.1"

as you can see the POST /user/oauth sometimes go to s1 sometimes go to s2 and sometimes to NOSRV.

sometimes it pass 10 times in a row and sometimes it fails 10 times in a row.

here is my config:

defaults
log     global
mode    http
option  httplog
option  dontlognull
timeout connect 10000
timeout client  50000
timeout server  50000
retries 3
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend local_server
bind *:8080 ssl crt /etc/letsencrypt/live/xxx/sslcert.pem
mode http
default_backend app

backend app
balance roundrobin
cookie SERVERID insert indirect nocache
http-send-name-header Host
server s1 172.26.13.134:3000 check cookie s1
server s2 172.26.13.134:3000 check cookie s2
http-request add-header X-Server %b

Edit:

i did redirect the request the a quick python server running on local host and still the problem persist.
so i did edit haproxy config file to have only server s1 127.0.0.1:8086 check cookie s1

and run the python script: python3 -m http.server 8086 –bind 127.0.0.1&

when i hit from postman i got like 3 succ on a row than a 503.