We have a Apache forward proxy sending requests to thousands of back end servers.
The servers are faulty, in that they require both the initial challenge request (without authentication) and the next request (with authentication) to be on the same http connection. We cannot get this fixed in any reasonable timescale.
So we would like the forward proxy to use the same connection, possibly via connection pooling?
Apache forward proxies , by default, closes the connection as soon as it gets a response. This is by design
“The default worker for forward proxying does not use connection
pooling in the naive sense. It closes each connection after each
request.What Ryujiro Shibuya was observing was that Apache signals it would
keep the connection open even in forward proxy mode, but then acually
closes the connection. We are discussing a fix to this, namely always
signalling "Connection: close" from the beginning for the default
forward and revere proxy workers. “
There is some suggestion it can be worked around:
"You can define explicit workers though (e.g. using ProxyPass for
reverse and as Rüdiger wrote likely also in forward proxy mode, which
then will use HTTP Keep-Alive (by default, depending on several
config options)"
But I don't know how to do this. Something with ProxySet, possibly? The issue with that is that I need to somehow specify the URLs, but this is a forward proxy – there are many possible origin servers and I cannot enumerate them up front.
How should we configure things to get this connection re-use?
Best Answer
(this should be a comment, but its a bit long)
And you've only just noticed its not working?
If you can handle the traffic for "thousands of servers" on a single Apache instance, it rather suggests something wrong with your architecture.
You can't rely on every component in the chain (i.e. not just your proxy) implementing keepalives in the way the back end servers seem to be expecting. So what you are attempting here is at best mitigation of an issue.
I would strongly urge you to fix the root cause here.
No. Connection pooling maintains a set of open connections to the origin server. It does not tie the client side connection to the origin server connection.
Without knowing a lot more about whats going on in the proxy server it's hard to advise what the right approach to this workaround is; while Apache makes a fabulous webserver, IMHO, there are better tools to use as proxies (ATS, haproxy, pound, squid, nginx, varnish.
From my knowledge of Apache httpd, I don't think its possible to do what you are trying to achieve.