Why server send RST packet to client after graceful close

rst

I 'm testing server stress test.
server OS and client OS is CentOS6.4.

client try to 3000 connect, one connection send a one http request, by load test tool, weighttp.
but, client get error. this error is connection reset by peer.
I captured packets by wireshark.

0.523605     10.0.0.2 -> 10.0.0.1     TCP 74 45763 > ddi-tcp-1 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=178982970 TSecr=0 WS=128
0.533484     10.0.0.1 -> 10.0.0.2     TCP 74 ddi-tcp-1 > 45763 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=178914097 TSecr=178982970 WS=128
0.589445     10.0.0.2 -> 10.0.0.1     TCP 66 45763 > ddi-tcp-1 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=178982993 TSecr=178914097
0.789282     10.0.0.2 -> 10.0.0.1     TCP 166 45763 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=178983235 TSecr=178914097
0.839889     10.0.0.1 -> 10.0.0.2     TCP 66 ddi-tcp-1 > 45763 [ACK] Seq=1 Ack=100 Win=14592 Len=0 TSval=178914363 TSecr=17898323
0.845196     10.0.0.1 -> 10.0.0.2     TCP 303 ddi-tcp-1 > 45763 [PSH, ACK] Seq=1 Ack=100 Win=14592 Len=237 TSval=178914374 TSecr=178983235
0.845203     10.0.0.1 -> 10.0.0.2     TCP 678 ddi-tcp-1 > 45763 [PSH, ACK] Seq=238 Ack=100 Win=14592 Len=612 TSval=178914374 TSecr=178983235
1.071341     10.0.0.1 -> 10.0.0.2     TCP 915 [TCP Retransmission] ddi-tcp-1 > 45763 [PSH, ACK] Seq=1 Ack=100 Win=14592 Len=849 TSval=178914640 TSecr=178983235
1.076500     10.0.0.2 -> 10.0.0.1     TCP 66 45763 > ddi-tcp-1 [ACK] Seq=100 Ack=850 Win=16384 Len=0 TSval=178983531 TSecr=178914640
1.076575     10.0.0.2 -> 10.0.0.1     TCP 66 45763 > ddi-tcp-1 [FIN, ACK] Seq=100 Ack=850 Win=16384 Len=0 TSval=178983531 TSecr=178914640
1.076844     10.0.0.1 -> 10.0.0.2     TCP 66 ddi-tcp-1 > 45763 [FIN, ACK] Seq=850 Ack=101 Win=14592 Len=0 TSval=178914649 TSecr=178983531
1.337245     10.0.0.2 -> 10.0.0.1     TCP 66 45763 > ddi-tcp-1 [ACK] Seq=101 Ack=851 Win=16384 Len=0 TSval=178983535 TSecr=178914649
1.463888     10.0.0.1 -> 10.0.0.2     TCP 66 [TCP Retransmission] ddi-tcp-1 > 45763 [FIN, ACK] Seq=850 Ack=101 Win=14592 Len=0 TSval=178914911 TSecr=178983531
1.471462     10.0.0.2 -> 10.0.0.1     TCP 66 [TCP Dup ACK 9057#1] 45763 > ddi-tcp-1 [ACK] Seq=101 Ack=851 Win=16384 Len=0 TSval=178983923 TSecr=178914649
1.586270     10.0.0.1 -> 10.0.0.2     TCP 54 ddi-tcp-1 > 45763 [RST] Seq=851 Win=0 Len=0

I don't understand.
Why server send RST packet to client after graceful close ?

I'm sorry for my bad English skill.

Thank you for replay.

I capturead packets on server side and client side.
server is very delay.

server packets

  8.537458     10.0.0.2 -> 10.0.0.1     TCP 74 51446 > ddi-tcp-1 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=720269 TSecr=0 WS=128
  8.537461     10.0.0.1 -> 10.0.0.2     TCP 74 ddi-tcp-1 > 51446 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=691010 TSecr=720269 WS=128
 10.279094     10.0.0.2 -> 10.0.0.1     TCP 165 51446 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=722038 TSecr=691010
 10.281834     10.0.0.1 -> 10.0.0.2     TCP 54 ddi-tcp-1 > 51446 [RST] Seq=1 Win=0 Len=0

client packets 

  6.959709     10.0.0.2 -> 10.0.0.1     TCP 74 51446 > ddi-tcp-1 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=720269 TSecr=0 WS=128
  6.964438     10.0.0.1 -> 10.0.0.2     TCP 74 ddi-tcp-1 > 51446 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=691010 TSecr=720269 WS=128
  7.000836     10.0.0.2 -> 10.0.0.1     TCP 66 51446 > ddi-tcp-1 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=720315 TSecr=691010
  7.002803     10.0.0.2 -> 10.0.0.1     TCP 165 51446 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=720315 TSecr=691010
  7.222562     10.0.0.2 -> 10.0.0.1     TCP 165 [TCP Retransmission] 51446 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=720562 TSecr=691010
  7.514462     10.0.0.1 -> 10.0.0.2     TCP 66 ddi-tcp-1 > 49507 [FIN, ACK] Seq=850 Ack=101 Win=14592 Len=0 TSval=691534 TSecr=720767
  7.703119     10.0.0.2 -> 10.0.0.1     TCP 165 [TCP Retransmission] 51446 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=721054 TSecr=691010
  8.686885     10.0.0.2 -> 10.0.0.1     TCP 165 [TCP Retransmission] 51446 > ddi-tcp-1 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=99 TSval=722038 TSecr=691010
  8.709193     10.0.0.1 -> 10.0.0.2     TCP 60 ddi-tcp-1 > 51446 [RST] Seq=1 Win=0 Len=0

I checked os log (/var/log/messaeges), but, I couldn't find that server reached limits.

I am trying to resolve porblem.
Thank you!

Best Answer

The close doesn't look very graceful on the server side due to the retransmission of the FIN,ACK @1.463888, as if it didn't see the ACK @1.337245

My best guess is the initial close ACK @1.337245 does eventually get processed after 1.463888 and the RST is for the subsequent ACK @1.471462

Try doing the capture on 10.0.0.1 at the same time to compare both side's of the connection and see when/if the packets appear on the server.

There also appears to be a significant delay to the clients initial close ACK @1.337245 which could also be a contributing factor. The clients previous ACK @1.076500 is after a retransmission as well. Maybe you are nearing the limit's of the client system? Try splitting the client requests across two or more boxes to remove that as a possible limiting factor.

Related Topic