We've previously had issues in our environment where it seemed we were reaching FreeBSDs max connection limit. We took action demonstrated by http://nginx.org/en/docs/freebsd_tuning.html and bumped up our connections limit to 500: kern.ipc.somaxconn: 500
We're still having issues where we expect to see additional connections from clients to this particular server and we are not, they're missing.
We don't know whether this is an application issue or a networking issue. But I was wondering if there is some other limit we're hitting now, maybe with max number of established connections?
Here's some graphs showing our connection behavior which looks strangely like it's limited at 300 connections:
Approximate number of missing commands on our server ^
EDIT:
Additional details about the application: The application is a tclsh (tcl shell) listening on a specific tcp port using the socket application to receive connections. Maybe this is some tcl-based limitation or a socket application limit?
Troubleshooting Details:
When I run nmap to basically "ping" the desired port over and over again
for i in {1..600}; do nmap -p 2069 serverIP; done
I seem to get the following with netstat:
netstat -Lan | grep 2069
tcp4 193/0/128 *.2069
tcp4 193/0/128 *.2069
tcp4 193/0/128 *.2069
tcp4 193/0/128 *.2069
This seems to mean that I am actually maxing out kern.ipc.somaxconn defaults. But we have already set that value to much higher than the default.
Even when I monitor established connections using:
netstat -an | grep 2069 | wc -l
I only get a grand total of 192 connections on 2069. Meaning it's not accepting more on that particular port.
Best Answer
Seems like it was actually an application limit. The process listening on the 2069 socket was built with a max number of 192 listen connections.
I'm assuming a lot of applications are built like with these limits in place and its likely that somaxconn just increases total allowable listening sockets vs what the application is actually built for.