Linux Server does not respond to TCP connections after some time running. How to analyse

linuxnetworkingtcpUbuntu

My Ubuntu 11.04 server on the internet has some strange behavior since a few days. It runs perfectly fine with some Java web applications. Then, suddenly it does not accept connections anymore. When I try to ssh or to http-connect my server I get no response, until I get timeout. But ping works perfectly. nmap also works:

Starting Nmap 5.21 ( http://nmap.org ) at 2011-08-29 10:52 CEST
Nmap scan report for ...
Host is up (0.020s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
53/tcp   open  domain
443/tcp  open  https
3000/tcp open  ppp
3128/tcp open  squid-http

After reboot, everything works again for some hours.

What could this be? Or how to analyse this problem?

Best Answer

This really does look like you are running out of memory, with no swap on the system. If a linux system runs out of memory, it cannot accept TCP connections anymore because the connection needs memory to be established. ICMP might not need anything since there is not state to maintain.

Check your memory settings everywhere, and make sure you do not allocate more than 70% of the total memory to the JVM (-Xms and -Xmx options).

Activate a swap if not yet done, you can create a basic swap file somewhere on the disk:

dd if=/dev/zero of=/mnt/swapfile bs=1M count=10240
mkswap /mnt/swapfile
swapon /mnt/swapfile

If after that your system hangs again, it's time for some low level monitoring.

Related Solutions

Linux – nmap shows opened port but netstat doesn’t

Are you on the same segment as the server in question? Portscanning via routers can give bogus results.

Linux – How to run a server on port 80 as a normal user on Linux

Short answer: you can't. Ports below 1024 can be opened only by root. As per comment - well, you can, using CAP_NET_BIND_SERVICE, but that approach, applied to java bin will make any java program to be run with this setting, which is undesirable, if not a security risk.

The long answer: you can redirect connections on port 80 to some other port you can open as normal user.

Run as root:

# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

As loopback devices (like localhost) do not use the prerouting rules, if you need to use localhost, etc., add this rule as well (thanks @Francesco):

# iptables -t nat -I OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-ports 8080

NOTE: The above solution is not well suited for multi-user systems, as any user can open port 8080 (or any other high port you decide to use), thus intercepting the traffic. (Credits to CesarB).

EDIT: as per comment question - to delete the above rule:

# iptables -t nat --line-numbers -n -L

This will output something like:

Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080 redir ports 8088
2    REDIRECT   tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 redir ports 8080

The rule you are interested in is nr. 2, so to delete it:

# iptables -t nat -D PREROUTING 2

Best Answer

Related Solutions

Linux – nmap shows opened port but netstat doesn’t

Linux – How to run a server on port 80 as a normal user on Linux

Related Topic