Ubuntu – Server not responding from outside the LAN

httpnetworkingUbuntu

I have an Ubuntu Server 14.04 running Apache that is behind a corporate firewall. This has been working fine for the last month but now the server is not responding to requests coming from outside the LAN (port forwarding done by the firewall).
Now my initial thought was to think something on apache had gone wrong but when I try to access the server using the internal address (192.168.8.10), everything works fine.
When I try to telnet from the local network:

$ telnet 192.168.8.10 80
Trying 192.168.8.10...
Connected to 192.168.8.10.
Escape character is '^]'.

Netstat on the server is giving me the following output:

$netstat -n -c | grep "192.168.8.19:80"
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV
tcp        0      0 192.168.8.10:80         192.168.8.250:49728     SYN_RECV

and when I visit http://192.168.8.10 in a browser, netstat produces:

$netstat -n -c | grep "192.168.8.19:80"
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     ESTABLISHED
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2
tcp6       0      0 192.168.8.10:80         192.168.8.252:63798     FIN_WAIT2

Now, when I try this from outside the LAN, telnet doesn't connect and the browser just times out.

$ telnet 78.xx.xx.245 80
Trying 78.xx.xx.245...

This just hangs indefinitely but netstat still acknowledges the SYN_RECV packets.

$ netstat -n -c | grep "192.168.8.19:80"
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV
tcp        0      0 192.168.8.10:80         194.xx.xx.62:52721      SYN_RECV

The browser test does not produce any output in netstat.

This then led me to think that the problem was with the port forwarding on the firewall. However when I tried forward port 80 to a Windows server, it works absolutely fine. When I changed port 80 to be forwarded back to the Ubuntu server, same problem. The fact that the telnet test actually reaches the server through the firewall indicates to me that the firewall is not the issue.

My IT department tell me that there have been no changes to the configuration of the firewall in the last few days and I haven't made any changes to the server. In fact, trying to forward port 80 to another Ubuntu desktop machine with apache running yielded the same results (hanging browser and telnet not connecting despite the netstat output).

Other ports are forwarded successfully to a Windows Server behind the firewall (and 80 worked when forwarded to the windows server) so I don't think the issue is with the ISP. It appears to solely be an issue with Ubuntu.

Has anyone got any ideas as to what this could be as it is a major problem for us now?

Best Answer

When a host can communicate with the LAN but not outside the LAN, the most likely misconfiguration to look for is an incorrect gateway IP address.

The symptoms if the host with a misconfigured gateway is the server side of a TCP connection will be that SYN packets are received and accepted, but SYN-ACK packets cannot be delivered due to not having a correct gateway address. This aligns with the symptoms described in your question.

A poor workaround would be to NAT the source IP of incoming connections at the gateway, which appears to be what your IT department has done. It's not a good idea because you lose the client IP address in your logfiles, and it potentially introduces additional connection tracking. It may appear to work just fine however, since the server no longer sends packets to the client IP address but rather to the gateway IP address which because it is on a directly attached network segment doesn't depend on the misconfigured default gateway.

The proper fix in this case is to configure the correct gateway IP address on the server and revert the workaround which was applied on the firewall.

If your Ubuntu server is configured with a static IP address, both IP address and gateway address are in /etc/network/interfaces. Changing that file takes effect at next reboot.

Related Topic