On a customer server running Apache 2.2 on Windows server 2012 we're noticing that from time to time, some requests to the server never finish. Using wireshark I've found a bunch of duplicate ACKs get sent to the server as soon as it starts answering the request and after a few seconds a couple of retransmissions are received from the server.
The network setup is really basic with a server and some clients connected to a switch using UTP cables.
I'm not really sure what to make of this. I'm considering asking them to try different cabling, switch and/or NIC's, but would like a second opinion on that.
1 0.000000000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 0.000742000 192.168.2.100 -> 192.168.1.103 TCP 66 81?52011 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
3 0.000782000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=1 Ack=1 Win=65536 Len=0
4 0.001646000 192.168.1.103 -> 192.168.2.100 HTTP 606 GET /symfony/web/app.php/legacy/mutaties/afroepen.php?afroepid=16250 HTTP/1.1
5 0.002353000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [ACK] Seq=1 Ack=553 Win=65536 Len=0
6 0.747171000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
7 0.747172000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
8 0.747246000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=2921 Win=65536 Len=0
9 0.747504000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
10 0.747507000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
11 0.747562000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=5841 Win=65536 Len=0
12 0.748241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
13 0.748242000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
14 0.748243000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
15 0.748244000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
16 0.748319000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=10221
17 0.748338000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#1] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=11681
18 0.748593000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
19 0.748594000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
20 0.748595000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
21 0.748624000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#2] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=13141
22 0.748643000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#3] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=14601
23 0.748654000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#4] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=16061
24 0.748965000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
25 0.748966000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
26 0.748967000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
27 0.748999000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#5] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=18981 SLE=8761 SRE=16061
28 0.749014000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#6] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=20441 SLE=8761 SRE=16061
29 0.749249000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
30 0.749250000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
31 0.749275000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#7] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=21901 SLE=8761 SRE=16061
32 0.749289000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#8] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=23361 SLE=8761 SRE=16061
33 0.749578000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Fast Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
34 0.749581000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
35 0.749907000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
36 0.749909000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
37 0.749910000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=16061 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
38 0.749939000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#9] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=26281 SLE=17521 SRE=23361 SLE=8761 SRE=16061
39 0.749958000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#10] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=27741 SLE=17521 SRE=23361 SLE=8761 SRE=16061
40 0.750240000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
41 0.750241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
42 0.750269000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#11] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=29201 SLE=17521 SRE=23361 SLE=8761 SRE=16061
43 0.750289000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#12] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=30661 SLE=17521 SRE=23361 SLE=8761 SRE=16061
44 0.750622000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
45 0.750624000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
46 0.750649000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#13] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
47 0.751016000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=23361 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
48 0.751017000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
49 0.751059000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#14] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=35041 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
50 0.751549000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
51 0.751551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
52 0.751567000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#15] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=36501 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
53 0.751582000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#16] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=37961 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
54 0.752392000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
55 0.752393000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
56 0.752394000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=32121 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
57 0.752396000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
58 0.752406000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#17] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
59 0.752433000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#18] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=42341 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
60 0.753551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
61 0.753553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
62 0.753571000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#19] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=43801 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
63 0.753588000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#20] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=45261 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
64 0.754550000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
65 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=39421 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
66 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
67 0.754572000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#21] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=46721 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
68 0.754610000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#22] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
69 1.055597000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
70 1.664583000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
71 2.867842000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
72 5.274379000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
73 10.071711000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
74 19.685995000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [RST, ACK] Seq=8761 Ack=553 Win=0 Len=0
75 19.686030000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#23] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
76 45.752325000 192.168.1.103 -> 192.168.2.100 TCP 55 [TCP Keep-Alive] [TCP Window Full] 52011?81 [ACK] Seq=552 Ack=7301 Win=65536 Len=1
77 58.686133000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
78 58.985830000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
79 59.585853000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
80 60.785894000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
81 63.185944000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
82 67.988211000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
83 77.582500000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [RST, ACK] Seq=554 Ack=7301 Win=0 Len=0
Best Answer
That log screams, to me, of either a cabling problem or a duplex mismatch between a device and the switch. I'd consider forcing the speed / duplex on the NICs on two devices (the server and a client) to 100Base-TX half-duplex and working up from there.
Shoddy patch cables, bad NIC drivers, or a failing switch could cause these behaviors, too. It's important to try to isolate the problem by taking a systematic approach-- changing one thing at a time and testing before making further changes.