Linux – Connections stuck in ESTABLISHED state

linuxnetworkingwindows

On a Debian machine I'm having several connections hanging for about 8 hours:

tcp6       0      0 192.168.1.35:56312      88.191.79.XXX:25        ESTABLISHED
tcp6       0      0 192.168.1.35:43352      88.191.79.XXX:25        ESTABLISHED
tcp6       0      0 192.168.1.35:56300      88.191.79.XXX:25        ESTABLISHED
tcp6       0      0 192.168.1.35:43350      88.191.79.XXX:25        ESTABLISHED

All connections are to the same remote mail server machine located in a different network. The remote machine actually doesn't see these connections. Tcpdump also shows no activity.

The client is Java app and has 4 threads blocked in socket read:

"MessageFeedbackQueueProc-Worker-202" prio=10 tid=0x00007f9c552f1800 nid=0x53d5 runnable [0x0000000043fb6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)

All these connections are happened in a small period of time, before and after which everything was/is working properly.

Interesting thing that the same happened on the other Windows machine with the same Java app in the same time period. In this case it disabled the app from processing new requests as all the threads became blocked by socket read (no timeout ever happened despite the dead remote end point). Another strange thing: all the threads on both machines became blocked after exactly same SMTP command (MAIL FROM) but in different time moments.

Why this could happen and why these connections don't time out?

Best Answer

I've seen something similar when the connection was being terminated at a firewall. Neither endpoint ever sees a FIN or RST so from their perspective the session is still active and in the Established state. Is it possible for you to look at the firewall logs on both ends to see if the connection is being terminated on either one?

It could also be an idle session timeout setting on the firewall on either end IF there's no data traversing the connection when the idle session timeout period expires.