Linux – Strange limit of 65536 max tcp incoming connections

linuxtcp

I have a Linux server with an Erlang application which shows an odd limit of exactly 65536 max tcp incoming connections.

The Erlang application is written with the Cowboy framework.

I have tuned the kernel parameters in this way:

/etc/sysctl.conf:

# Increase system file descriptor limit
fs.file-max = 300000

# Discourage Linux from swapping idle processes to disk (default = 60)
vm.swappiness = 10

# Increase Linux autotuning TCP buffer limits
# Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Make room for more TIME_WAIT sockets due to more clients,
# and allow them to be reused if we run out of sockets
# Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10

# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0

# Disable source routing and redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0

# Log packets with impossible addresses for security
net.ipv4.conf.all.log_martians = 1

/etc/security/limits.conf:

*         soft    nofile          300000
*         hard    nofile          300000

I also checked the maximum number of Erlang processes, but it doesn't seem the limiting factor:

1> erlang:system_info(process_limit).
262144

The test connections are made by 4 client machines from 4 different IP addresses (so it isn't a problem with client limits) and after an initial authentication, the connections are left idle, so server CPU utilization is below 50% and memory utilization is also below 35%.

EDIT: I'm not sure if this is important: the Erlang server application is listening on port 8000 and I get it to listen on port 80 too with this iptables rule:

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000

What else should I look into to remove the 65536 connections limit?

Best Answer

Check how is coded your application. There is a limit of 65535 port, and those in that range: 0-1024 are reserved.

If your application use ephemeral's port; from RFC-6056

  1. Ephemeral Ports

2.1. Traditional Ephemeral Port Range

The Internet Assigned Numbers Authority (IANA) assigns the unique
parameters and values used in protocols developed by the Internet
Engineering Task Force (IETF), including well-known ports [IANA].
IANA has reserved the following use of the 16-bit port range of TCP
and UDP:

o The Well-Known Ports, 0 through 1023.

o The Registered Ports, 1024 through 49151

o The Dynamic and/or Private Ports, 49152 through 65535

The dynamic port range defined by IANA consists of the 49152-65535 range, and is meant for the selection of ephemeral ports.

In last step ask the vendor if it's a hardcoded value. (an int_16)

Related Topic