TCP Dup ACK linux kernel 3.2

apache-2.4httpslinux-kerneltcp

I am downloading very slowly on some servers we host when using HTTPS on debian 8.1 kernel 3.16.0-4-amd64 Apache 2.4.10-10

The architecture is :
CLIENT X.X.X.X => NAT ON FIREWALL: Y.Y.Y.Y => HTTP(S) WEB SERVER LAN IP:10.254.248.101

Test with HTTPS : 

wget --no-check-certificate https://Y.Y.Y.Y/test/debian-8.5.0-amd64-netinst.iso
debian-8.5.0-amd64-netinst.iso.5            0%[      ] 632.00K  15.5KB/s   eta 4h 30m

When running tcpdump trace on server (trace below), the client X.X.X.X ask 10.254.248.101 to resend packet many times.
The server wait almost 4 seconds to resend the packet asked by the client.
This is why I am downloading so slowly on the server. What I don't understand is why the server take a so long time to resend the packet.
Is there something new in linux Kernel 3 for TCP ? Something new in Apache 2.4 ? Someting that FIREWALL is not able to manage ?

Note that, We don't have this problem in these cases :

  • When using HTTP (without TLS) (test below).
  • When clients come from
    ISP that have longer latency (200 to 250 ms) to our datacenter.
  • With same architecture, same SSL certificate but older linux kernel 2.6
    and older apache 2.2.16-6.
  • When we put WEB SERVER before firewall set
    with pulic IP : CLIENT X.X.X.X => Y.Y.Y.Y.

Test with HTTP :

wget http://Y.Y.Y.Y//test/debian-8.5.0-amd64-netinst.iso
debian-8.5.0-amd64-netinst.iso.6           35%[=============================>   ]  86.53M  1.34MB/s   eta 2m 2s

Trace of the serveur :

366 7.930488    X.X.X.X 10.254.248.101  66  TCP 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604
369 7.931293    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#1] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546404710
370 7.933094    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#2] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546406098
371 7.933984    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#3] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546407486
372 7.934031    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#4] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668402 TSecr=44657604 SLE=3546403322 SRE=3546408874
373 7.935729    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#5] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546410262
374 7.936154    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#6] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546411650
375 7.937320    X.X.X.X 10.254.248.101  78  TCP [TCP Dup ACK 366#7] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546403322 SRE=3546413038
376 7.938613    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#8] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546415814 SLE=3546403322 SRE=3546413038
377 7.940425    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#9] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546417202 SLE=3546403322 SRE=3546413038
378 7.941661    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#10] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546418590 SLE=3546403322 SRE=3546413038
379 7.942561    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#11] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546419978 SLE=3546403322 SRE=3546413038
380 7.943155    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#12] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546421366 SLE=3546403322 SRE=3546413038
381 7.945145    X.X.X.X 10.254.248.101  86  TCP [TCP Dup ACK 366#13] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
382 7.945992    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#14] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546424142 SRE=3546425530 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
383 7.946974    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#15] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668403 TSecr=44657604 SLE=3546424142 SRE=3546426918 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
384 7.948476    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#16] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546428306 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
385 7.949704    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#17] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546429694 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
386 7.950628    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#18] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546431082 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
387 7.951890    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#19] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546432470 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
388 7.953365    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#20] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546433858 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
389 7.954028    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#21] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546435246 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
390 7.955327    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#22] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546436634 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
391 7.956139    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#23] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546438022 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
392 7.957504    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#24] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546439410 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
393 7.958271    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#25] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546440798 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
394 7.963017    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#26] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668404 TSecr=44657604 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754 SLE=3546403322 SRE=3546413038
395 7.963080    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#27] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546444962 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
396 7.963091    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#28] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546446350 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
397 7.963127    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#29] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546447738 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
398 7.963912    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#30] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546449126 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
399 7.965634    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#31] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546450514 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
400 7.966653    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#32] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546451902 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
401 7.967402    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#33] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546453290 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
402 7.968499    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#34] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546454678 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
403 7.969984    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#35] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546456066 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
404 7.971168    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#36] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668405 TSecr=44657604 SLE=3546443574 SRE=3546457454 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
405 7.972479    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#37] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186 SLE=3546414426 SRE=3546422754
406 7.973633    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#38] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546461618 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186
407 7.974609    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#39] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546463006 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186
408 7.975828    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#40] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842 SLE=3546424142 SRE=3546442186
409 7.976604    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#41] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546467170 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
410 7.977806    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#42] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546468558 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
411 7.979200    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#43] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546469946 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
412 7.982792    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#44] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546471334 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
413 7.982817    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#45] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546472722 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
414 7.983331    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#46] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546474110 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
415 7.983631    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#47] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668406 TSecr=44657604 SLE=3546465782 SRE=3546475498 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
416 7.984371    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#48] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546476886 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
417 7.986238    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#49] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546478274 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
418 7.987165    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#50] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546479662 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
419 7.988157    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#51] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668407 TSecr=44657604 SLE=3546465782 SRE=3546481050 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
421 8.113745    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#52] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668420 TSecr=44657604 SLE=3546465782 SRE=3546482438 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
424 8.150664    X.X.X.X 10.254.248.101  60  TCP 65259  >  443 [ACK] Seq=1251 Ack=2197 Win=65700 Len=0
428 8.373938    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#54] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668446 TSecr=44657604 SLE=3546465782 SRE=3546485214 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
434 8.504768    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#55] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668459 TSecr=44657604 SLE=3546465782 SRE=3546486602 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
436 8.630589    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#56] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668472 TSecr=44657604 SLE=3546465782 SRE=3546487990 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
439 8.759603    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#57] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668485 TSecr=44657604 SLE=3546465782 SRE=3546489378 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
441 8.886688    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#58] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668498 TSecr=44657604 SLE=3546465782 SRE=3546490766 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
443 9.015359    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#59] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668510 TSecr=44657604 SLE=3546465782 SRE=3546492154 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
445 9.141981    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#60] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668523 TSecr=44657604 SLE=3546465782 SRE=3546493542 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
447 9.271283    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#61] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668536 TSecr=44657604 SLE=3546465782 SRE=3546494930 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
449 9.398242    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#62] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668549 TSecr=44657604 SLE=3546465782 SRE=3546496318 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
451 9.525714    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#63] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668562 TSecr=44657604 SLE=3546465782 SRE=3546497706 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
453 9.653846    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#64] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668574 TSecr=44657604 SLE=3546465782 SRE=3546499094 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
455 9.782376    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#65] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668587 TSecr=44657604 SLE=3546465782 SRE=3546500482 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
457 9.909855    X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#66] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668600 TSecr=44657604 SLE=3546465782 SRE=3546501870 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
459 10.038509   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#67] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668613 TSecr=44657604 SLE=3546465782 SRE=3546503258 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
461 10.166752   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#68] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668626 TSecr=44657604 SLE=3546465782 SRE=3546504646 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
467 10.303645   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#69] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668639 TSecr=44657604 SLE=3546465782 SRE=3546506034 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
474 10.430023   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#70] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668652 TSecr=44657604 SLE=3546465782 SRE=3546507422 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
476 10.557951   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#71] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668665 TSecr=44657604 SLE=3546465782 SRE=3546508810 SLE=3546460230 SRE=3546464394 SLE=3546443574 SRE=3546458842
478 10.689920   X.X.X.X 10.254.248.101  94  TCP [TCP Dup ACK 366#72] 33426  >  443 [ACK] Seq=831 Ack=223341 Win=248576 Len=0 TSval=94668678 TSecr=44657604 SLE=3546507422 SRE=3546508810 SLE=3546465782 SRE=3546508810 SLE=3546460230 SRE=3546464394
482 11.045636   X.X.X.X 62.61.231.82    1454    TCP [TCP Retransmission] 443  >  33426 [ACK] Seq=223341 Ack=831 Win=31104 Len=1388 TSval=44658396 TSecr=94668678

Best Answer

Thanks for sending those captures over.

The Problem

Your throughput issues appear to be caused by a buggy implementation of TCP Sequence Number randomization. I have seen this in the past on Cisco ASAs.

To give a bit of background, it was observed in the past that some TCP implementations did not use enough randomness when choosing an Initial Sequence Number (ISN) which made it easier for attackers to manipulate TCP connections by making educated guesses at what the Sequence number would be.

To attempt to fix this issue, some firewall providers implemented a feature called TCP sequence number randomization, which rewrites the Sequence number (SEQ) to a more random value, when it sees TCP packets flowing through the firewall. Unfortunately some implementations of this feature are a bit buggy and do not account for TCPs Selective Acknowledgement (SACK) feature.

You can see Sequence Number randomization in action in your trace. Look at the SYN/ACK packet from the server (packet #51 server capture), where you can see that the ISN chosen is 2847541373. However look at the same SYN/ACK packet when it is received on the client side (packet #8 client capture), the ISN has been changed to 2098751282!

This behavior is fine up until the point that packet loss is experienced on the network.

On the client side, look at the first Duplicate Acknowledgement (Dup ACK) at packet 259. You can see that a SACK block has been set covering bytes 2098977399-2098978787. This packet effectively tells the server, I'm waiting on packet with SEQ 2098974623, however I have received 2098977399-2098978787 so you don't need to send those again.

Now, if you look at the same Dup ACK as it is received on the server side (#369), you can see the ACK number has been correctly converted by the firewall (2098974623 > 2847764714), however the SACK block hasn't and still shows 2098977399-2098978787!

When a Dup ACK is received with an invalid SACK block, the Dup ACK is ignored.

As a result, you lose out on the ability to use Fast Retransmission (retransmit after 3 duplicate ACKs received) and rely solely on Retranmission Timeouts. This is really, really bad for performance and will reduce your throughput substantially.

So what can you do?

You can investigate whether TCP Sequence Number randomisation is still required for your purposes and if not, consider testing with it disabled. Perhaps this issue has been resolved in a newer firmware?

You could also turn off the TCP SACK option on your server(s) to prevent clients from using SACK in the first place /proc/sys/net/ipv4/tcp_sack however please note that SACK is meant to be used to improve TCP performance and the actual issue is with the firewalls (buggy) implementation of Sequence number randomization. Turning off SACK will mean that Dup ACK's from clients will no longer be ignored and the connection will be able to recover from loss a lot quicker. Throughput should go up.

Related Topic