I'm currently setting up an HAproxy machine to act as a reverse proxy for a couple of websites. As the different machines are not physically connected I'm using OpenVPN to create a VPN between the machines.
Technically, everything is working. However, my problem is that requests that I pipe through the proxy are extremely slow. I get the complete response(s) eventually, but it takes up to 5 minutes to receive something like a Jenkins dashboard.
I tried a simple static HTML page served via NGINX as well as some REST API implemented in go – the result is always the same: I get the data eventually but it takes a ridiculous amount of time.
Here's my HAproxy config:
global
log /var/run/log local0 info
log /var/run/log local0 notice
daemon
maxconn 8000
tune.ssl.default-dh-param 2048
user nobody
group nobody
defaults
log global
option httplog
option dontlognull
mode http
timeout connect 5s
timeout client 1min
timeout server 1min
option forwardfor
option http-server-close
errorfile 400 /usr/local/etc/haproxy/errorfiles/400.http
errorfile 403 /usr/local/etc/haproxy/errorfiles/403.http
errorfile 408 /usr/local/etc/haproxy/errorfiles/408.http
errorfile 500 /usr/local/etc/haproxy/errorfiles/500.http
errorfile 502 /usr/local/etc/haproxy/errorfiles/502.http
errorfile 503 /usr/local/etc/haproxy/errorfiles/503.http
errorfile 504 /usr/local/etc/haproxy/errorfiles/504.http
frontend http-in
bind *:80
bind *:443 ssl crt /usr/local/etc/haproxy/certs/foo.my.org.pem
mode http
use_backend jenkins if { hdr(host) -i foo.my.org }
use_backend test if { hdr(host) -i bar.my.org }
default_backend test
backend jenkins
server jenkins1 <vpn_ip>:8180
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
reqrep ^([^\ :]*)\ /(.*) \1\ /\2
acl response-is-redirect res.hdr(Location) -m found
rspirep ^Location:\ (http)://<vpn_ip>:8180/(.*) Location:\ https://foo.my.org:443/\2 if response-is-redirect
backend test
server web01 <vpn_ip>:80
Additional information:
- All involved machines are located in data centers and feature a 1G/1G internet connection
- All machines run FreeBSD 11 64-Bit
- OpenVPN version 2.4.4 on all machines
- HAproxy version 1.7.9
- OpenVPN runs in TCP mode
- The HAproxy backend config for Jenkins is taken as provided from the Jenkins documentation but as mentioned the same problem exists with a bare static HTML content webserver and a REST API web service.
And here's some HAproxy log when accessing the Jenkins site:
Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:60243 [27/Feb/2018:01:32:24.093] http-in~ jenkins/jenkins1 0/0/13/134/161 302 153 - - ---- 5/5/0/0/0 0/0 "GET /jenkins HTTP/1.1"
Feb 27 01:32:24 hostname haproxy[5539]: 213.144.130.227:19404 [27/Feb/2018:01:32:24.255] http-in~ jenkins/jenkins1 0/0/25/174/212 200 4492 - - ---- 5/5/0/0/0 0/0 "GET /jenkins/ HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:16321 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/13/30/54 200 8560 - - ---- 6/6/4/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/datasource/datasource-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:54637 [27/Feb/2018:01:32:25.330] http-in~ jenkins/jenkins1 0/0/27/29/58 200 7585 - - ---- 6/6/3/4/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/autocomplete/autocomplete-min.js HTTP/1.1"Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:59247 [27/Feb/2018:01:32:25.361] http-in~ jenkins/jenkins1 0/0/25/16/51 200 9602 - - ---- 6/6/2/3/0 0/0 "GET /jenkins/static/aeed77bb/jsbundles/page-init.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:40637 [27/Feb/2018:01:32:25.332] http-in~ jenkins/jenkins1 0/0/38/18/81 200 16212 - - ---- 6/6/1/1/0 0/0 "GET /jenkins/static/aeed77bb/scripts/yui/menu/menu-min.js HTTP/1.1"
Feb 27 01:32:25 hostname haproxy[5539]: 213.144.130.227:10976 [27/Feb/2018:01:32:25.333] http-in~ jenkins/jenkins1 0/0/37/30/95 200 29110 - - ---- 6/6/0/0/0 0/0 "GET /jenkins/static/aeed77bb/scripts/hudson-behavior.js HTTP/1.1"
I don't see anything else in the log files. No errors or similar.
Can this be due to the OpenVPN?
Edit 1: In the mean time I tested this without OpenVPN by directly using the web server's public IP address in the HAproxy configuration. The result is exactly the same.
Best Answer
Your log doesn't include enough information to explain the 5 minute wait -- it spans less than 2 seconds -- but here is what jumps out:
This is not appropriate with a back-end on the other side of a WAN link. If you actually have a need for this feature, it needs to be on an HAProxy with extremely low latency (e.g., local) to the back-end server.
Note the large amount of jitter in your
Tc
timer values. This is the third value in the log field that looks like this:So, in the first entry,
0/0/13/134/161
, theTc
value is 13 ms.Tc
is the time required to establish the connection to the back-end, so what we see here is that your round-trip jitter is pretty much out of control, jumping in one case to almost three times this value. Assuming this isn't caused by insufficient resources on the back-end server, this is likely to be a consequence of using OpenVPN in TCP mode -- possibly, but not necessarily, indicating some packet loss.You're tunneling TCP (HTTP) inside TCP (OpenVPN), which is usually fine for non-interactive applications and applications where jitter is unimportant, but not otherwise. (See, for example, Why TCP over TCP is a Bad Idea for discussion of one of the potential meltdown scenarios, though not sufficient reason to avoid this configuration entirely on a clean network.) Through what I assume to be no fault of OpenVPN, TCP tunnels do not have the same consistent latency as UDP tunnels. Ping through the tunnel and you should see this jitter imparted to the ping response times.
Change the tunnel to UDP and the issue should disappear... but note that the significance and performance impact of this jitter -- assuming I am correct that it is a side effect of using a TCP tunnel -- is probably exaggerated by using
option http-server-close
, which requires that each request establish a new connection between the HAProxy and the back-end.If your OpenVPN tunnels are compressed using
comp-lzo
, then consider disabling that, as well, since poteblntial cost savings in transport may be outweighed by the overhead.