How to debug linux networking: Address already in use

linux-networkingnetworking

I have a Slackware linux box where I cannot start any service that listens on one particular port on localhost. By using strace I found out that the error happens on the bind() call, and the error is EADDRINUSE (Address already in use):

bind(3, {sa_family=AF_INET, sin_port=htons(874), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)

This happens with any process I try to start listening on that port, so it is not related to the process itself. The above strace output comes from the command strace -ff nc -l -p 874 -s 127.0.0.1.

So, this suggests there is a process already listening on localhost port 874. However, I can't seem to find it. The following commands all return nothing:

netstat -aplunt | grep :874
netstat -na | grep :874
lsof -i :874
lsof -i tcp | grep 874
fuser 874/tcp
socklist | grep 874
iptables -t filter -S | grep 874
iptables -t nat -S | grep 874
iptables -t mangle -S | grep 874
conntrack -L | grep 874

If I try to listen on 0.0.0.0:874 it fails with the same error. Listening on one of the IP addresses configured on a nic works OK, and listening to 127.0.0.2:874 also works OK. Listening on a different port works fine, also on 127.0.0.1 or 0.0.0.0.

So, now I am curious. How can I find out why the network stack returns EADDRINUSE here? What other things could I look at, or what other commands can I run to get more information?

Additional info:

Kernel 4.1.31.
Selinux is not used here.
Trying to connect to 127.0.0.1 with telnet returns "Connection refused"
I'm running the commands as root

Best Answer

If your host is an NFS client, it may be using source port 874 for an NFS mount. I suspect that because the connection does not originate from userspace it may not be visible to the tools you've used so far.

Consider one of the following:

Adjust the sysctls sunrpc.min_resvport and sunrpc.max_resvport (default 665 and 1023) to change the range of source ports that the NFS client uses
Use a listening port outside of this range
Use the noresvport option on the NFS mount to use the non-privileged range (may have security implications)

1.

Strictly IPv4 speaking, is port exhaustion actually possible?

Yes. Take, for example, a load balancing router sending all connections to a NAT IP address. This is likely to happen when you have many SRC IPs connecting to the bottleneck of a single DST IP.

This means that your webserver could have a bunch of connections like:

root@buglab:~# netstat -pnt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 173.200.1.18:80      10.100.1.100:49923        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.200.1.200:10155        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:14400        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.10.1.10:50652        ESTABLISHED 13939/nginx: worker
tcp        0      0 173.200.1.18:80      10.20.1.20:57554        ESTABLISHED 13939/nginx: worker

and that's perfectly fine. However, if all the 'Foreign Addresses' were the same, this can cause an issue (e.g. 'big router that performs NAT <---> server with one IP address').

If I had to postulate as to why ephemeral port exhaustion isn't a common issue, I'd suggest it's because every port requires a listening service and enough resources to respond -- another resource (memory, cpu) is normally a bottleneck first.

However, I've personally come across a few port exhaustion issues when working at a load balancing company.

2. Why can a used port present an issue for a listening service?

"Which allows the use of ephemeral ports from 1024-65535, that if I have services that bind on port 3306 (mySQL, for example), they will sometimes fail to start because the port is in use."

The mySQL server cannot bind to that port if it's in use - say by localhost:3306 or on all interfaces. For example, see the 0.0.0.0:80 line in the following netstat output?

root@buglab:~# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      PID/Program name
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      964/php-fpm.conf)
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1660/mysqld     
tcp        0      0 0.0.0.0:842             0.0.0.0:*               LISTEN      1317/inetd      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      13938/nginx

That means that port 80 is listening across all interfaces local to the server. If another process holds port 80 before my nginx server starts, nginx will not be able to take control of that port and will likely fail its startup procedure.

Normally, port 3306 is fine because listening services have pre-defined ports (or ranges) that are requested from the host machine - e.g. port 80 and 443 for webservers.

How is Elastic Beanstalk forwarding traffic on port 80 to the app at port 8080 without nginx/apache

It's a NAT rule.

iptables -L -t nat

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Thanks to @slims_s from reddit.

Best Answer

Related Solutions

Linux networking port exhaustion

1.

2. Why can a used port present an issue for a listening service?

How is Elastic Beanstalk forwarding traffic on port 80 to the app at port 8080 without nginx/apache

Related Topic