Linux – Figuring out the reason for a “Connection timed out” error

centosfirewalllinuxnetscaler

For the last 24 hours two independent servers are having a hard time connecting to our web server. They are trying to open a port 80 connection.

They are both receiving "connection timed out" errors. These servers are in two different countries. I'm waiting for tracert's from these servers.

But in the meantime, I'm thinking the reason has to be with our own server or the netscaler in front it. Hosting company insists there is no problem with the netscaler.

Where can I check on our centos box to trace refused connections? Are there any logs I can look into?

UPDATE:
Make that three independent servers. It turns out Paypal cannot send Instant Payment Notices as well.
All three services are trying to call separate pages on our server.
We've used Paypal Sandbox to repeat the error and now we can continously test it. We tested this on other websites. Paypal sends a http request as follows:

Content-Type: application/x-www-form-urlencoded  
Content-Length: 699  

Unfortunately our server does not receive this http request. We tried this on another server behind the same netscaler. It does not receive the http request as well.
I think the netscaler is blocking the connection for some reason.

Why would the netscaler block this connection? What should I look into?

UPDATE-2:
Any http request without "user-agent" info seems to be blocked at the netscaler level.

Best Answer

Instead of looking into server logs, first make sure if the HTTP request arrives at your server or not via ;

tcpdump -nni any host 111.111.111.111 and port 80

As 111.111.111.111 being the remote IP. If you don't see anything once they initiate the connection, problem is at somewhere else. You may not see refused connections in a web server log file if it is somehow blocked at a lower layer.