I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max
. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout
seconds for more connections to become available. The conntrack
table is what the kernel uses to track the state of connections so once it's full, the kernel starts dropping packets and printing this in the log:
Jun 2 20:39:14 XXXX-XXX kernel: ip_conntrack: table full, dropping packet.
The next step was getting the kernel to recycle all those connections in the TIME_WAIT
state rather than dropping packets. I could get that to happen either by turning on tcp_tw_recycle
or increasing ip_conntrack_max
to be larger than the number of local ports made available for connections by ip_local_port_range
. I guess once the kernel is out of local ports it starts recycling connections. This uses more memory tracking connections but it seems like the better solution than turning on tcp_tw_recycle
since the docs imply that that is dangerous.
With this configuration I can run ab all day and never run out of connections:
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 32768 61000
The tcp_max_orphans
setting didn't have any effect on my tests and I don't know why. I would think it would close the connections in TIME_WAIT
state once there were 8192 of them but it doesn't do that for me.
You should know that Windows XP (and probably other versions) has an internal wrapper for FTP connections (the purpose of this is to try to allow PORT command to complete successfully, even behind a firewall or a router).
This wrapper intercepts any connection to any host on port 21, so it can monitor it and try to open the incoming port of a PORT command issued by the client.
This wrapper also has a side effect: as it intercepts any connection to a port 21, it sends a signal that the connection has been established to the software, which will see the connection as established, but the connection is really established only to Windows's internal wrapper.
The wrapper then tries to open the connection to the real host, and if it timeouts, then it sends a signal to the software that the connection has been lost. The software will see the connection as lost.
Summing this up, the software believes a connection has been successfully established, then lost, but no real connection has been established.
So, in your case, what happens: you run nmap. Nmap tries to connect to your server on port 21. Windows's wrapper intercepts the connection. Nmap "thinks" it is connected to your server (but it's only connected to the wrapper), and reports the port as opened.
You can confirm this by typing in a command line:
ftp 4.3.2.1
You'll see:
C:>ftp 4.3.2.1
Connected to 4.3.2.1.
Connection closed by foreign host.
You can try any valid IP, ftp will always connect, and disconnect shortly after, whereas it should report "Connection timed out".
I never saw any documentation about this. After many investigation, I discovered this strange behavior, and after more investigation, discovered why it is here.
Well, the conclusion of this (big) answer is that the port 21 of your server is definitely closed, as netstat reports, and nmap is fooled by this behaviour.
Best Answer
-l
will list listening ports,-p
will also display the process,-n
will show port numbers instead of names. Add-t
to only show TCP ports.