Linux – Tomcat not getting data from TCP recv-Q, hanging

linuxtcptomcat

I have a program which is essentially a tomcat server with postgreSQL DB running on CentOS Linux 6.3. When a client submits a larger amount of data (~10MB) to the server, which should be received on TCP 9090, the client seems to hang and not get a response from the server. I saw nothing significant in the logs, all I could find was that the connection existed on port 9090 and that there was about 3.5 MB sitting in the recv-Q in netstat's output. Is this likely a system or application issue?

This is what strace shows:
9089 futex(0x7fdd80862528, FUTEX_WAKE_PRIVATE, 1) = 0
9089 futex(0x7fdd81e145a4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1374870092, 226655000}, ffffffff <unfinished ...>
9085 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
9085 futex(0x7fdd88004728, FUTEX_WAKE_PRIVATE, 1) = 0
9085 futex(0x7fdd94013464, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1374870092, 226812000}, ffffffff <unfinished ...>
9071 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
9071 futex(0x7fde00092c28, FUTEX_WAKE_PRIVATE, 1) = 0
9071 futex(0x40b62e64, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1374870087, 319631000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
9071 futex(0x7fde00092c28, FUTEX_WAKE_PRIVATE, 1) = 0
9071 futex(0x40b62e64, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1374870087, 369779000}, ffffffff

Best Answer

This ended up being a low memory issue for the java process. Adding RAM and tuning the DB resolved it. Java memory issues? Shocking, huh?