I'm setting up ELK stacks with loadbalancing. Haproxy works fine for ES and Kibana but I'm having issues with Logstash.
Here's the haproxy configuration:
frontend logstash
bind 0.0.0.0:5000
mode tcp
option tcpka
option tcplog
log global
acl epa hdr_beg(host) -i epa-log
acl tgops hdr_beg(host) -i tgops-log
use_backend epa_log if epa
use_backend tgops_log if tgops
backend epa_log
mode tcp
server elk01 elk01.example.org:5001 check
server elk02 elk02.example.org:5001 check
server elk03 elk03.example.org:5001 check
backend tgops_log
mode tcp
server elk01 elk01.example.org:5002 check
server elk02 elk02.example.org:5002 check
server elk03 elk03.example.org:5002 check
As you can see, im trying to have 1 frontend serving port 5000 and redirecting to a different port based on the url of the source.
If I do a netcat directly to the server it works perfectly
nc elk01 5001 -vv < /var/log/httpd/error_log
But if I use the haproxy frontend, it fails with a broken pipe error.
nc tgops-log 5000 -vv < /var/log/httpd/error_log
Ncat: Version 6.40 ( http://nmap.org/ncat )
libnsock nsi_new2(): nsi_new (IOD #1)
libnsock nsock_connect_tcp(): TCP connection requested to 10.129.10.2:5000 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [10.129.10.2:5000]
Ncat: Connected to 10.129.10.2:5000.
libnsock nsi_new2(): nsi_new (IOD #2)
libnsock nsock_read(): Read request from IOD #1 [10.129.10.2:5000] (timeout: -1ms) EID 18
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 26 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 35 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 42
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 42 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 51 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 58
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 58 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 67 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 74
libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [10.129.10.2:5000]
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 74 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 83 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 90
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 90 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 99 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 106
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 106 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 115 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 122
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 122 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 131 [10.129.10.2:5000]
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 138
libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 138 [peer unspecified] (8192 bytes)
libnsock nsock_trace_handler_callback(): Callback: WRITE ERROR [Broken pipe (32)] for EID 147 [10.129.10.2:5000]
Ncat: Broken pipe.
If I check the TCP logs, I find that it somehow won't reach the backend, throws a NOSRV
logstash logstash/<NOSRV> -1/-1/0 0 SC 15/0/0/0/0 0/0n 10.129.10.82:50724
I've tried different HAProxy options, like clitcpka
, balance source, using only 1 backend and more but none resolves this.
Best Answer
You can't use
mode tcp
and match headers.mode tcp
means don't treat the connection as HTTP and just pass the raw bytes through.