How to i find out why the web server is refusing traffic from the web proxy

connectionPROXYtroubleshooting

I have a bit of an odd problem that I can't seem to figure out. I will preface this question with the fact that this exact setup was working just fine, and for some reason has stopped working properly.

I have a web proxy to ensure users at one of my clients offices have their web traffic filtered before going to the internet. This proxy (Quinto Labs QLProxy) sits in the DMZ of that network. Also in that DMZ is a web server that hosts several small applications.

The flow of traffic is as follows (for web browsing)

USER (10.0.0.0/24) --> Web Proxy (172.16.0.6) --> Outside world

The flow of traffic when a user tries to access a website that's hosted on the web server in the DMZ is as follows

USER (10.0.0.0/24) --> Web Proxy (172.16.0.6) --> Web Server (172.16.0.7)

Recently, the web server has started refusing traffic to all but one of the websites hosted on the web server. The error message on the browser is The system returned: (110) Connection timed out

Settings for the web site that works are seemingly identical to that which don't work. When browsing to a website on the web server that doesn't work I see SYN_SENT with a netstat -ant directed to the web server. I also see SYN SENT on my router from the proxy to the web server. The DNS resolves fine for both the working website and those that don't work. I have entries in the /etc/hosts file on the proxy server to ensure their accuracy. I have tried using the private IP address and the public IP address for those entries and the results seem the same.

Dropping the firewall completely on the web server doesn't change anything and the connection is still refused.

Does anyone know why this might be happening, or is able to tell me how I might figure out why this is happening?

EDIT :

  • As far as I know nothing has changed to cause this issue – I cannot think of anything that has changed that might have caused it.
  • The iptables rules are empty on the Proxy machine
  • If I telnet into the web server from the proxy using the internal IP address it works, however if I telnet in via the external IP address (using port 80) it does not. Is this a NAT issue then??
  • Web server is Server 2012
  • No firewall changes have been made to either the proxy machine or the web server.

Best Answer

You don't describe what your NAT device is but it sounds, to me, like that NAT device isn't capable of doing hairpin (or loopback) NAT.

It is perplexing as to how this might have changed suddenly. I can think of a number of possibilities. Perhaps you recently changed something w/ the NAT device's configuration. It's also possible that you've been using split-horizon DNS to return the private addresses of the web site (as opposed to the public address) and these records were recently removed. Maybe your proxy server was using an internal DNS server w/ split-horizon records and has recently been reconfigured to use Internet DNS.

Running a sniffer on the web server and confirming that the SYN packets from the proxy aren't ever making it to the web server would tend to put the blame squarely on your NAT device.

Related Topic