Ssl – use HAProxy’s new ‘capture’ feature to save the remote address in a TCP frontend, and use it as the `X-Forwarded-For` header in an HTTP backend

haproxypacket-capturessl

Using HAProxy 1.6 and a clever hack, I now have an HAProxy tcp mode frontend, that detects if the browser is capable of SNI, and based on that, routes to a strongly ciphered SSL termination backend, or a weaker one. This ensures A+ grading on SSL labs, while still allowing all browsers except IE6 to use SSL.

Here is my config. It has some template variables in it that should be self-explanatory, but aren't in areas relevant to my question:

frontend https_incoming
 bind 0.0.0.0:443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 tcp-request content accept if { req.ssl_hello_type 1 }
 use_backend https_strong if { req.ssl_sni -m end .transloadit.com }
 default_backend https_weak

backend https_strong
 mode tcp
 option tcplog
 server https_strong 127.0.0.1:1665

frontend https_strong
 bind 127.0.0.1:1665 ssl crt ${DM_ROOT_DIR}/envs/ssl/haproxy-dh2048.pem no-sslv3 no-tls-tickets ciphers ${strongCiphers}
 mode http
 option httplog
 option httpclose
 option forwardfor if-none except 127.0.0.1
 http-response add-header Strict-Transport-Security max-age=31536000
 reqadd X-Forwarded-Proto:\ https
 reqadd FRONT_END_HTTPS:\ on
 use_backend http_incoming

backend https_weak
 mode tcp
 option tcplog
 server https_weak 127.0.0.1:1667

frontend https_weak
 bind 127.0.0.1:1667 ssl crt ${DM_ROOT_DIR}/envs/ssl/haproxy.pem no-sslv3 ciphers ${weakCiphers}
 mode http
 option httplog
 option httpclose
 option forwardfor if-none except 127.0.0.1
 http-response add-header Strict-Transport-Security max-age=31536000
 reqadd X-Forwarded-Proto:\ https
 reqadd FRONT_END_HTTPS:\ on
 use_backend http_incoming

Problem: the https_incoming frontend knows the Client IP, but since it is in mode tcp, it cannot save this information in a mode http X-Forwarded-For header. option forwardfor is not valid in TCP mode.

From another question on serverfault
I already found that I could use:

  • LVS
  • PROXY protocol

So that the X-Forwarded-For header isn't even needed anymore as from what I understand, in the case of LVS: packets are spoofed so the source becomes the Client IP, and in the case of PROXY: packets are encapsulated to carry the Client IP.

These both seem like they could work. LVS however seems quite a heart-surgery for us that could have side-effects, and PROXY has the downside that proxies/application upstream/downstream, might not be fully compatible yet.

I was really hoping for something more lightweight, and that's when I found the new "Capture" feature of HAProxy 1.6 as it mentions:

you can declare capture slots, store data in it and use it at any time during a session.

it goes on to show the following example:

defaults 
 mode http

frontend f_myapp
 bind :9001
 declare capture request len 32 # id=0 to store Host header
 declare capture request len 64 # id=1 to store User-Agent header
 http-request capture req.hdr(Host) id 0
 http-request capture req.hdr(User-Agent) id 1
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[capture.req.hdr(0)]
 http-response set-header Your-User-Agent %[capture.req.hdr(1)]
 server s1 10.0.0.3:4444 check

It appears to me, information is stored in a frontend, and then later used in a backend, so perhaps I can take the Client IP in TCP mode, save it, and use that later down the line, maybe like so:

http-response set-header X-Forwarded-For %[capture.req.hdr(0)]

I've looked at the capture docs and there it seems capture is more oriented at http mode headers, but then I have also seen a mailing list conversation successfully demonstrating the use of a tcp-request capture.

I've tried several things, among which:

tcp-request capture req.hdr(RemoteAddr) id 0
# or
tcp-request content capture req.hdr(RemoteHost) id 0

But as you can see, I haven't got a clue what the syntax should be and under which key this information would be available, nor can I find it in the (I think) relevant documentation.

Questions: Would it be possible to capture the Client IP in TCP mode, and later down the line, write this information into the X-Forwarded-For header in HTTP mode? If so, what would be the syntax for this?

Best Answer

To answer my own question, this does not seem possible, as the traffic 'leaves' HAProxy here:

       TCP                             HTTP
frontend->backend (->leaving->) frontend->backend

So the context is lost and the capture cannot be preserved. Instead, as "PiBa-NL" suggested on IRC at #haproxy on Freenode yesterday:

[5:29pm] PiBa-NL: kvz, use proxy-protocol between back and front
[5:54pm] kvz: PiBa-NL: Thanks, does this mean my app also needs to understand 
         the proxy protocol, or will it be 'stripped' once it reaches the 
         backend. I don't think my node.js server could handle it without  
         significant changes to its stack
[6:07pm] kvz: Or let me rephrase: could I enable the proxy protocol on the first 
         frontend, then 'unwrap' it in the second frontend, taking the client ip 
         and putting it into the http header - so that my app would not have to 
         be proxy protocol compatible, and it would just be means to carry the 
         client ip from first frontend to the second?
[6:49pm] PiBa-NL: kvz, the last part you can still use the x-forwarded-for header
[6:50pm] PiBa-NL: but between haproxy backend and frontend you would use the 
         proxyprotocol to make the second frontent 'know' the original client ip
[6:50pm] PiBa-NL: server https_strong 127.0.0.1:1665 send-proxy
[6:50pm] PiBa-NL: bind 127.0.0.1:1665 ssl crt .... accept-proxy
[6:52pm] PiBa-NL: the second frontend can then still use the  
         'option forwardfor', and it should insert the wanted header
[6:53pm] PiBa-NL: so basically 'yes' 

This means the PROXY protocol is only used to glue the two frontends together, encapsulating the Cient IP, but the second frontend unwraps it and saves it in the X-Forwarded-For header via option forwardfor, so that its backend can send a PROXY-protocol-less request to my app server, meaning I do not have to worry about compatibility issues up/downstream.