HAProxy CentOS7 – Unable to Switch to Transparent Mode

I'm currently in the process of setting up HAProxy in transparent mode because it's supposed to balance between nodes of a software that doesn't support the PROXY protocol, wants to terminate SSL itself and does its own IP-based security.

I've been through every search result and guide on how to set up transparent mode that I could find, and while it seems very straightforward, it just doesn't appear to work at all.

Base is a CentOS 7 minimal install, HAProxy is running as root. It's the default gateway for the backend servers, has TPROXY loaded and has the necessary firewall rules and networking set up.

Using tcpdump, I can see that all traffic to and from the backend servers is passing through HAProxy, but the source IP in the packets simply isn't replaced. Communication with the backend and load balancing work perfectly as far as I can see, by the way.

haproxy.cfg:

global
    log         127.0.0.1 local0 debug
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    daemon
    stats socket /var/lib/haproxy/stats uid hatop gid hatop mode 0600

defaults
    mode                    tcp
    log                     global
    option                  tcplog
#   option                  dontlognull
    option                  redispatch
    retries                 3
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000
    balance                 leastconn

frontend f_general_plain
    bind 10.1.1.20:80 transparent       # HTTP
    bind 10.1.1.20:465 transparent      # SMTP
    bind 10.1.1.20:587 transparent      # Submission
    bind 10.1.1.20:143 transparent      # IMAP
    bind 10.1.1.20:5229 transparent     # Groupware
    bind 10.1.1.20:32002 transparent    
    bind 10.1.1.20:5222 transparent     # XMPP
    bind 10.1.1.20:5223 transparent 
    default_backend b_general_plain

backend b_general_plain
    stick-table type ip size 1m expire 10h
    stick on src
    source 0.0.0.0 usesrc clientip
    server iw1 10.1.1.16 
    server iw2 10.1.1.17 

frontend f_general_ssl
    bind 10.1.1.20:443 transparent      # HTTPS
    bind 10.1.1.20:465 transparent      # SMTPS
    bind 10.1.1.20:993 transparent      # IMAPS
    default_backend b_general_ssl

backend b_general_ssl
    stick on src table b_general_plain
    source 0.0.0.0 usesrc clientip
    server iw1 10.1.1.16
    server iw2 10.1.1.17 

frontend f_smtp
    bind 10.1.1.20:25 transparent
    default_backend b_smtp

backend b_smtp
    stick on src table b_general_plain
    option smtpchk EHLO balancer.example.com
    source 0.0.0.0 usesrc clientip
    server iw1 10.1.1.16:25 check
    server iw2 10.1.1.17:25 check

/etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 1

/etc/firewalld/direct.xml – firewalld barely has anything else set, just opening the necessary ports etc. According to iptables -t mangle -vL, the DIVERT chain does get used.

<?xml version="1.0" encoding="utf-8"?>
<direct>
<chain ipv="ipv4" table="mangle" chain="DIVERT"/>
<rule  ipv="ipv4" table="mangle" chain="PREROUTING" priority="0">-p tcp -m socket -j DIVERT</rule>
<rule  ipv="ipv4" table="mangle" chain="DIVERT" priority="0">-j MARK --set-mark 1</rule>
<rule  ipv="ipv4" table="mangle" chain="DIVERT" priority="1">-j ACCEPT</rule>
</direct>

ip rule:

0:  from all lookup local 
32765:  from all fwmark 0x1 lookup 100 
32766:  from all lookup main 
32767:  from all lookup default

ip route show table 100:

local default dev lo scope host

lsmod | grep -i tproxy

xt_TPROXY              17327  0 
nf_defrag_ipv6         35104  3 xt_socket,xt_TPROXY,nf_conntrack_ipv6
nf_defrag_ipv4         12729  3 xt_socket,xt_TPROXY,nf_conntrack_ipv4

uname -a

Linux balancer 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

haproxy -vv

HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
  OPTIONS = USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.