We're seeing a large number of packets being dropped due to being Out-of-Order. The numbers are large enough that it may be having an impact on network performance.
We've isolated it down to a few internal IPs and the connections seem to be hitting Google's servers on 443. The two computers shown in the Gist below both run Chrome and the Hangouts extension; the same as everyone else in the company. One is a Mac, one is a PC.
I'm turning up nothing in trying to figure out what the heck is going on. Can anyone help me shed some light on this or what to do next?
house200-fw01# sh asp drop
Frame drop:
No route to host (no-route) 56
Flow is denied by configured rule (acl-drop) 1459
First TCP packet not SYN (tcp-not-syn) 37
TCP failed 3 way handshake (tcp-3whs-failed) 351
TCP RST/FIN out of order (tcp-rstfin-ooo) 530
TCP packet SEQ past window (tcp-seq-past-win) 167
TCP Out-of-Order packet buffer full (tcp-buffer-full) 15421
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 1950
TCP RST/SYN in window (tcp-rst-syn-in-win) 2
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 1874
Dropped pending packets in a closed socket (np-socket-closed) 4
- 848: 11:15:37.578903 173.194.46.86.443 > 33.33.33.33.29847
- PAT Global 33.33.33.33(29847) Local 10.55.55.109(52294)
- 827: 11:15:05.322386 207.238.18.142.443 > 33.33.33.33.23232
- PAT Global 33.33.33.33(23232) Local 10.55.55.79(49555)
- capture drop type asp-drop tcp-dup-in-queue buffer 3344556
1: 11:07:38.267442 173.194.57.120.443 > 33.33.33.33.49558: . 195566822:195568202(1380) ack 598914623 win 386
2: 11:07:38.267564 173.194.57.120.443 > 33.33.33.33.49558: . 195570962:195572342(1380) ack 598914623 win 386
3: 11:07:38.267671 173.194.57.120.443 > 33.33.33.33.49558: . 195573722:195575102(1380) ack 598914623 win 386
4: 11:07:38.267793 173.194.57.120.443 > 33.33.33.33.49558: . 195576482:195577862(1380) ack 598914623 win 386
5: 11:07:38.267915 173.194.57.120.443 > 33.33.33.33.49558: . 195577862:195579242(1380) ack 598914623 win 386
6: 11:07:38.268144 173.194.57.120.443 > 33.33.33.33.49558: . 195583382:195584762(1380) ack 598914623 win 386
7: 11:07:38.268250 173.194.57.120.443 > 33.33.33.33.49558: . 195587522:195588902(1380) ack 598914623 win 386
8: 11:07:38.268403 173.194.57.120.443 > 33.33.33.33.49558: . 195590282:195591662(1380) ack 598914623 win 386
9: 11:07:38.268601 173.194.57.120.443 > 33.33.33.33.49558: . 195594422:195595802(1380) ack 598914623 win 386
10: 11:07:38.268723 173.194.57.120.443 > 33.33.33.33.49558: . 195597182:195598562(1380) ack 598914623 win 386
11: 11:07:38.292511 173.194.57.120.443 > 33.33.33.33.49558: . 195599942:195601322(1380) ack 598914623 win 386
12: 11:07:38.292862 173.194.57.120.443 > 33.33.33.33.49558: . 195604082:195605462(1380) ack 598914623 win 386
13: 11:07:38.292984 173.194.57.120.443 > 33.33.33.33.49558: . 195605462:195606842(1380) ack 598914623 win 386
14: 11:07:38.293090 173.194.57.120.443 > 33.33.33.33.49558: . 195606842:195608222(1380) ack 598914623 win 386
15: 11:07:38.293212 173.194.57.120.443 > 33.33.33.33.49558: . 195608222:195609602(1380) ack 598914623 win 386
16: 11:07:38.293335 173.194.57.120.443 > 33.33.33.33.49558: . 195609602:195610982(1380) ack 598914623 win 386
17: 11:07:38.293441 173.194.57.120.443 > 33.33.33.33.49558: . 195610982:195612362(1380) ack 598914623 win 386
18: 11:07:38.293563 173.194.57.120.443 > 33.33.33.33.49558: . 195612362:195613742(1380) ack 598914623 win 386
19: 11:07:38.293685 173.194.57.120.443 > 33.33.33.33.49558: . 195613742:195615122(1380) ack 598914623 win 386
20: 11:07:38.293792 173.194.57.120.443 > 33.33.33.33.49558: . 195615122:195616502(1380) ack 598914623 win 386
21: 11:07:38.293914 173.194.57.120.443 > 33.33.33.33.49558: . 195616502:195617882(1380) ack 598914623 win 386
22: 11:07:38.294494 173.194.57.120.443 > 33.33.33.33.49558: . 195623402:195624782(1380) ack 598914623 win 386
23: 11:07:38.294616 173.194.57.120.443 > 33.33.33.33.49558: . 195624782:195626162(1380) ack 598914623 win 386
- capture drop type asp-drop tcp-buffer-timeout buffer 3344556
1: 11:10:36.636762 173.194.57.120.443 > 33.33.33.33.7417: . 3709341327:3709342707(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
2: 11:10:36.636884 173.194.57.120.443 > 33.33.33.33.7417: . 3709339947:3709341327(1380) ack 2239501518 win 556
3: 11:10:36.636975 173.194.57.120.443 > 33.33.33.33.7417: . 3709338567:3709339947(1380) ack 2239501518 win 556
4: 11:10:36.637097 173.194.57.120.443 > 33.33.33.33.7417: . 3709337187:3709338567(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
5: 11:10:36.637219 173.194.57.120.443 > 33.33.33.33.7417: . 3709330287:3709331667(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
6: 11:10:36.637326 173.194.57.120.443 > 33.33.33.33.7417: . 3709328907:3709330287(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
7: 11:10:36.637448 173.194.57.120.443 > 33.33.33.33.7417: . 3709327527:3709328907(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
8: 11:10:36.637570 173.194.57.120.443 > 33.33.33.33.7417: . 3709326147:3709327527(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
9: 11:10:36.638562 173.194.57.120.443 > 33.33.33.33.7417: . 3709324767:3709326147(1380) ack 2239501518 win 556
10: 11:10:36.638730 173.194.57.120.443 > 33.33.33.33.7417: . 3709323387:3709324767(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
11: 11:10:36.638837 173.194.57.120.443 > 33.33.33.33.7417: . 3709322007:3709323387(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
12: 11:10:36.638959 173.194.57.120.443 > 33.33.33.33.7417: . 3709320627:3709322007(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
13: 11:10:36.727028 173.194.57.120.443 > 33.33.33.33.7417: . 3709559367:3709560747(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
14: 11:10:36.727150 173.194.57.120.443 > 33.33.33.33.7417: . 3709557987:3709559367(1380) ack 2239501518 win 556 Drop-reason: (tcp-buffer-timeout) TCP Out-of-Order packet buffer timeout
15: 11:10:36.727257 173.194.57.120.443 > 33.33.33.33.7417: . 3709551087:3709552467(1380) ack 2239501518 win 556
Best Answer
We had the same issue but for a different protocol.
While using IOS version 8.x (cannot remember the exact release), there was an issue with the Inspects that would mess up the order in wich it would relay the packets.
Do you have HTTP/HTTPS inspects enabled?
If so, you can create an ACL and "exclude" one or two machines from the inspects just for them just for testing purposes.