Python reports socket in use, netstat and others claim its not

netstatnetworkingsockettcp

We have a strange socket issue with a RHES3 box:

Python 2.4.1 (#1, Jul  5 2005, 19:17:11)
[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-52)]
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket()
>>> s.bind(('localhost',12351))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 1, in bind
socket.error: (98, 'Address already in use')

This seems normal, lets see what has that socket:

# netstat -untap | grep 12351

{no output}

# grep 12351 /proc/net/tcp

{no output}

# lsof | grep 12351

{no output}

# fuser -n tcp 12351

{no output, repeating the python test fails again}

# nc localhost 12351

{no output}

# nmap localhost 12351

{shows port closed}

Other high ports work fine (eg 12352 works)

Is there something magic about this port? Is there somewhere else I can look? Where does python find out that socket is in use that netstat doesnt know about? Any other way I can find out what/if that socket is?

Best Answer

Not sure if I had the same problem but i'll put what I found here.

I had a python server process that was bound to 8081 and Apache proxying to it using the reverse proxy module.

Looking in ps showed that the process was running:

UID    PID  PPID  C STIME TTY          TIME CMD
xxxx 31896 31681  2 08:51 ?        00:00:03  \_ python /opt/xxxx/install/xxxx/xxxx/bin/content_server.py localhost:8081

Telnet worked fine:

$ telnet localhost 8081
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

However greping for the port number in netstat didn't show anything bound to that port:

$ netstat -a | grep 8081
$

Though if you check the process id in netstat you might see a line like this with "tproxy" as the port number:

$ netstat -a -p | grep 31896
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp        0      0 MELLAGFIN01:tproxy  *:*  LISTEN  31896/python

Someone else can probably confirm, but i'm assuming this is some sort of kernel level tcp proxy?