Linux – Fixing Huge Amount of TIME_WAIT Connections

debianlinuxnetstatnetworkingtcpip

Okay, this is creeping me out – I see about 1500-2500 of these:

root@wherever:# netstat

Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 localhost:60930         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60934         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60941         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60947         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60962         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60969         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60998         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60802         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60823         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60876         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60886         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60898         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60897         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60905         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60918         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60921         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60673         localhost:sunrpc        TIME_WAIT  
tcp        0      0 localhost:60680         localhost:sunrpc        TIME_WAIT  
[etc...]

root@wherever:# netstat | grep 'TIME_WAIT' |wc -l
1942

That number is changing rapidly.

I do have a pretty tight iptables config so I have no idea what can cause this. any ideas?

Thanks,

Tamas

Edit: Output of 'netstat -anp':

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:60968         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60972         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60976         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60981         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60980         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60983         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60999         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60809         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60834         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60872         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60896         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60919         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60710         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60745         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60765         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60772         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60558         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60564         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60600         127.0.0.1:111           TIME_WAIT   -               
tcp        0      0 127.0.0.1:60624         127.0.0.1:111           TIME_WAIT   -

Best Answer

EDIT: tcp_fin_timeout DOES NOT control TIME_WAIT duration, it is hardcoded at 60s

As mentioned by others, having some connections in TIME_WAIT is a normal part of the TCP connection. You can see the interval by examining /proc/sys/net/ipv4/tcp_fin_timeout:

[root@host ~]# cat /proc/sys/net/ipv4/tcp_fin_timeout
60

And change it by modifying that value:

[root@dev admin]# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

Or permanently by adding it to /etc/sysctl.conf

net.ipv4.tcp_fin_timeout=30

Also, if you don't use the RPC service or NFS, you can just turn it off:

/etc/init.d/nfsd stop

And turn it off completely

chkconfig nfsd off

Related Solutions

How to Turn Off FTP for Enhanced Security on Debian

You should know that Windows XP (and probably other versions) has an internal wrapper for FTP connections (the purpose of this is to try to allow PORT command to complete successfully, even behind a firewall or a router).

This wrapper intercepts any connection to any host on port 21, so it can monitor it and try to open the incoming port of a PORT command issued by the client.

This wrapper also has a side effect: as it intercepts any connection to a port 21, it sends a signal that the connection has been established to the software, which will see the connection as established, but the connection is really established only to Windows's internal wrapper.

The wrapper then tries to open the connection to the real host, and if it timeouts, then it sends a signal to the software that the connection has been lost. The software will see the connection as lost.

Summing this up, the software believes a connection has been successfully established, then lost, but no real connection has been established.

So, in your case, what happens: you run nmap. Nmap tries to connect to your server on port 21. Windows's wrapper intercepts the connection. Nmap "thinks" it is connected to your server (but it's only connected to the wrapper), and reports the port as opened.

You can confirm this by typing in a command line:

ftp 4.3.2.1

You'll see: C:>ftp 4.3.2.1

Connected to 4.3.2.1.

Connection closed by foreign host.

You can try any valid IP, ftp will always connect, and disconnect shortly after, whereas it should report "Connection timed out".

I never saw any documentation about this. After many investigation, I discovered this strange behavior, and after more investigation, discovered why it is here.

Well, the conclusion of this (big) answer is that the port 21 of your server is definitely closed, as netstat reports, and nmap is fooled by this behaviour.

Linux port not accessible from remote machine

My guess is it's a firewall issue. To diagnose, either switch to root or use sudo to execute these commands described below:

Firstly, service iptables status will show you your current rules. Depending on how esoteric your firewall rules are, you may need to add them into the main question to debug but essentially you're looking for a line that accepts TCP port 8800 prior to a line that rejects all connections.

e.g.

Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source       destination
...
14   ACCEPT     tcp  --  0.0.0.0/0    0.0.0.0/0      state NEW tcp dpt:8800
15   REJECT     all  --  0.0.0.0/0    0.0.0.0/0      reject-with icmp-host-prohibited

If you don't have a REJECT line on your input chain, then the firewall is probably already turned off.

One way to get around this is obviously to turn off the firewall (service iptables stop) although this defeats the entire purpose of having a firewall.

If you need to add a new firewall rule, you have a couple of options:

Use the excellent Firewall Configuration tool (system-config-firewall) that ships with Fedora and can be found at the System->Administration->Firewall menu item in Gnome.
On the command line, insert a new rule before the REJECT rule by executing the following (uses numbers from my example above): iptables -I INPUT 15 -m state --state NEW -m tcp -p tcp --dport 8800 -j ACCEPT. NOTE that these changes are not permanent and will not persist beyond the next reboot.
Manually edit the /etc/sysconfig/iptables file and add a rule (-A INPUT -m state --state NEW -m tcp -p tcp --dport 8800 -j ACCEPT) to the firewall prior to the REJECT rule. Cycle the firewall service to pick up the changes via a service iptables restart. These changes will be applied every time the firewall service is started.

Best Answer

Related Solutions

How to Turn Off FTP for Enhanced Security on Debian

Linux port not accessible from remote machine

Related Topic