Linux – How to debug dropped SYN-ACK packets

linuxnetworkingtcp

I have 2 load balancers in front of 4 other server machines doing roughly a combined 5-6000 requests per second. I've noticed that every few minutes, one the load balancers gets a connection timeout connecting to one of the server machines. As a percentage of total connections, this is of course a very low failure rate, however, they are in the same datacenter and I can't see why I would ever get a connection timeout (60 seconds, fwiw).

I managed to get a packet capture one of the server machines when this happens and it looks like SYN-ACK packets are not reaching the load balancer.

19:33:51.023518 IP 10.36.213.2.58011 > 10.36.213.36.8005: Flags [S], seq 1726133855, win 5840, options [mss 1460,sackOK,TS val 14377140 ecr 0,nop,wscale 7], length 0
19:33:51.023534 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15334821 ecr 14377140,nop,wscale 7], length 0
19:33:54.026705 IP 10.36.213.2.58011 > 10.36.213.36.8005: Flags [S], seq 1726133855, win 5840, options [mss 1460,sackOK,TS val 14377440 ecr 0,nop,wscale 7], length 0
19:33:54.026712 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15335121 ecr 14377140,nop,wscale 7], length 0
19:33:54.620425 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15335181 ecr 14377140,nop,wscale 7], length 0
19:34:00.026709 IP 10.36.213.2.58011 > 10.36.213.36.8005: Flags [S], seq 1726133855, win 5840, options [mss 1460,sackOK,TS val 14378040 ecr 0,nop,wscale 7], length 0
19:34:00.026726 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15335721 ecr 14377140,nop,wscale 7], length 0
19:34:00.620436 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15335781 ecr 14377140,nop,wscale 7], length 0
19:34:12.620430 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15336981 ecr 14377140,nop,wscale 7], length 0
19:34:36.820432 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15339401 ecr 14377140,nop,wscale 7], length 0
19:35:24.820433 IP 10.36.213.36.8005 > 10.36.213.2.58011: Flags [S.], seq 806746037, ack 1726133856, win 5792, options [mss 1460,sackOK,TS val 15344201 ecr 14377140,nop,wscale 7], length 0

In this capture, 10.36.213.2 is the load balancer and 10.36.213.36 is the server machine.

My guess is that the chance that all those packets are just randomly being dropped is low… more likely they're being rejected in some way. So, the question is: how would I got about finding out why those packets aren't getting through?

Best Answer

I saw a similar problem (granted that was Windows) where the servers were doing some sort of stupid auto tuning of the TCP packet sizes and the Cisco Load Ballancers didn't support the larger packet size so they were rejecting them. I've got no idea if *nix supports this sort of stupidity but if it does that might be the problem.

Related Topic