Ldap – openldap with haproxy – (ldap_result() failed: Can’t contact LDAP server)

I'm having an issue with openldap proxied via haproxy. The authentication works perfectly fine on CentOS (7.5.1804) and Debian (9.4). The only problem is that I'm getting errors in syslog and authlog:

CentOS:
nslcd[10082]: [9f6e5f] ldap_result() failed: Can't contact LDAP server

Debian:
nscd: nss_ldap: reconnected to LDAP server ldap://haproxy.exemple.com after 1 attempt
The whole problem dissapears when I change the LDAP endpoint directly to any ldap server from the configuration below:

global
  log           127.0.0.1 local3
  pidfile       /var/run/haproxy.pid
  chroot        /var/lib/haproxy
  maxconn       8192
  user          haproxy
  group         haproxy
  daemon
  stats socket /var/lib/haproxy/stats
  tune.ssl.default-dh-param 2048


frontend ldap_service_front
  mode                  tcp
  bind                  10.0.0.2:389
  description           LDAP Service
  option                socket-stats
  option                tcpka
  timeout client        10s
  default_backend       ldap_service_back

backend ldap_service_back
  server                ldap01 10.0.0.3:389 check fall 3 rise 5 inter 5000 weight 10
  server                ldap02 10.0.0.4:389 check fall 3 rise 5 inter 5000 weight 10
  server                ldap03 10.0.0.5:389 check fall 3 rise 5 inter 5000 weight 10
  server                ldap04 10.0.0.6:389 check fall 3 rise 5 inter 5000 weight 10

mode              tcp
  balance           leastconn
  stick-table       type ip size 200k expire 30m
  timeout server    12s
  timeout connect   10s
  option            tcpka
  option            tcp-check
  tcp-check         connect port 389
  tcp-check         send-binary 300c0201
  tcp-check         send-binary 01
  tcp-check         send-binary 6007
  tcp-check         send-binary 0201
  tcp-check         send-binary 03
  tcp-check         send-binary 04008000
  tcp-check         expect binary 0a0100
  tcp-check         send-binary 30050201034200

frontend ldaps_service_front
  mode                  tcp
  bind                  10.0.0.2:636 ssl crt /etc/haproxy/ssl/chain.pem no-sslv3 no-tlsv10
  description           LDAPS Service
  option                socket-stats
  option                tcpka
  timeout client        10s
  default_backend       ldaps_service_back

backend ldaps_service_back
  server                ldap01 10.0.0.3:636 check ssl verify none fall 3 rise 5 inter 5000 weight 10
  server                ldap02 10.0.0.4:636 check ssl verify none fall 3 rise 5 inter 5000 weight 10
  server                ldap03 10.0.0.5:636 check ssl verify none fall 3 rise 5 inter 5000 weight 10
  server                ldap04 10.0.0.6:636 check ssl verify none fall 3 rise 5 inter 5000 weight 10

mode                  tcp
  balance               leastconn
  timeout server        12s
  timeout connect       10s
  option                tcpka
  option                tcp-check
  tcp-check             connect port 636 ssl
  tcp-check             send-binary 300c0201
  tcp-check             send-binary 01
  tcp-check             send-binary 6007
  tcp-check             send-binary 0201
  tcp-check             send-binary 03
  tcp-check             send-binary 04008000
  tcp-check             expect binary 0a0100
  tcp-check             send-binary 30050201034200

So, my question is: did I miss something in the haproxy configuration? I was trying to set up the idle_timelimit in nslcd.conf, but it didn't help at all.

More details that might help:

haproxy: HA-Proxy version 1.8.12-1~bpo9+1
openldap: slapd 2.4.44
nslcd: nss-pam-ldapd 0.8.13