High number of TCP RST packets

centos6tcp

Whilst looking into some transient network issues, I have noticed what seems to be a high number of RST packets received:

Tcp:
40091950 active connections openings
44754733 passive connection openings
2840116 failed connection attempts
49571981 connection resets received
30748407 resets sent

Can anyone comment on whether this would indicate a problem? It seems extraordinarily high, but I may just be misinterpreting the results. From what I have read an RST is sent when a TCP connection does not receive ACK for data it is sending.

We are running several Centos 6.5 servers, Webapps running in tomcat, balanced behind httpd. Each tomcat will make many short lived connections to each other, so this may be a result of that?

I have done some analysis, the RST packets are not coming from the router or from 1 specific host, but seem to be coming from all hosts.

Best Answer

TCP is a state machine. A TCP connection exists on both participants in the conversation, and both participants must agree on the current state of the connection.

A TCP Reset usually occurs when a system receives data which doesn't agree with its view of the connection.

Wikipedia has a great TCP State Diagram explaining all possible states and valid transitions between those states.

As for your issue, without seeing the traffic it's hard to tell, however it's somewhat common to see Resets on teardown when one end has closed the connection but the other end still had data buffered or in-flight to send.

Imagine the following

HostA             HostB
      ESTABLISHED
Data ------------>       1. here is some data
     <------------ ACK   2. i have received your data
Data ------------>       3. here is some data
     <------------ ACK   4. i have received your data
Data ------>             5. (data hasn't reached HostB yet)
     <------------ FIN   6. we're done here, close this connection
Data        ----->       7. (data reaches HostB)
     <------------ RST   8. i didn't expect this!

At 6, HostB sent a FIN so HostB's socket entered FIN_WAIT1 and expected an ACK from HostA.

However, instead of the expected ACK, HostB got more data. Sending data to a socket in FIN_WAIT1 is not valid, it doesn't agree with what HostB's state machine says is a next valid step. Something's gone wrong, so a Reset is sent to end the TCP conversation altogether.

Related Topic