Linux Networking – Troubleshooting Packet Drops

packetloss

I've run dropwatch and this is the result I got:

dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
39 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
36 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
3 drops at skb_release_data+10e (0xffffffff8157bf3e)
2 drops at tcp_v4_do_rcv+80 (0xffffffff815f8f70)
2 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
30 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
31 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
20 drops at unix_dgram_sendmsg+4f8 (0xffffffff81646a38)
5 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
2 drops at tcp_v4_do_rcv+80 (0xffffffff815f8f70)
19 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
23 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
2 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
2 drops at skb_release_data+10e (0xffffffff8157bf3e)
11 drops at unix_dgram_sendmsg+4f8 (0xffffffff81646a38)
57 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
49 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
5 drops at skb_release_data+10e (0xffffffff8157bf3e)
5 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
1 drops at skb_queue_purge+18 (0xffffffff8157c0a8)
4 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
4 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
3 drops at tcp_v4_do_rcv+80 (0xffffffff815f8f70)
3 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
10 drops at skb_release_data+10e (0xffffffff8157bf3e)
38 drops at unix_dgram_sendmsg+4f8 (0xffffffff81646a38)
29 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
28 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
1 drops at tcp_v6_rcv+87 (0xffffffff81677ff7)
2 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
1 drops at tcp_v4_do_rcv+80 (0xffffffff815f8f70)
1 drops at skb_release_data+10e (0xffffffff8157bf3e)
17 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
14 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
1 drops at skb_release_data+10e (0xffffffff8157bf3e)
1 drops at tcp_v4_rcv+87 (0xffffffff815fa087)
5 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
10 drops at skb_release_data+10e (0xffffffff8157bf3e)
2 drops at unix_dgram_sendmsg+4f8 (0xffffffff81646a38)
4 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
20 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
22 drops at sk_stream_kill_queues+50 (0xffffffff81583970)
2 drops at skb_release_data+10e (0xffffffff8157bf3e)
48 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)
53 drops at sk_stream_kill_queues+50 (0xffffffff81583970)

From now on I'm stuck. I've checked that tcp_rcv_state_process and sk_stream_kill_queues are functions in Linux Kernel but I don't know by what they are controlled. I've jumped on this issue because in my node some application timeout in a expected manner.

Any advice how could I go on?

Best Answer

In order to progress you need to install kernel debug modules and 'elfutils' package. On Centos 7:

#debuginfo-install kernel
#yum install elfutils

After that you can find source code position in kernel, that corresponds to the address from dropwatch. For example you have "57 drops at tcp_rcv_state_process+1b6 (0xffffffff815eeda6)"

#eu-addr2line -f -k 0xffffffff815eeda6
tcp_rcv_state_process
net/ipv4/tcp_input.c:5834

In this case it will drop data from tcp packet if it is a SYN packet.

Related Solutions

ISA Server dropping packets as it believes they are spoofed

I strongly suspect that your scanning problem has nothing to do with the ISA firewall, and more to do with SMB protocol security restrictions on the server computer where you're attempting to target the scan. I suspect this because, primarily, I'm guessing that the server you're attempting to send the scans to has a 192.168.16.0/24 IP address and, as such, communication between the scanner and the server won't need to move through the ISA Server.

The dropped packet is almost certainly a red herring.

The packet you're seeing dropped there is a NetBIOS name service-related broadcast. These are normal packets generated by implementations of the Microsoft file and print sharing protocol, SMB. This makes me wonder if you have a layer 2 connection between the "outside" network segment and your "inside" network segment and broadcasts from the "inside" are also showing up on the "outside" segment, too. Running a sniffer on the "outside" segment would tell you that quickly and easily. Is "the modem" attached directly to the ISA Server computer, or do you have it plugged into a switch/hub that might have other connections?

Most crappy copier/MFP devices that I've worked with require SMB signing to be disabled on the server computer to enable "scan to folder" functionality. There's some discussion about this in reference to Ricoh machines here on Server Fault. I'd strongly suspect this is your problem.

Linux – How passively monitor for tcp packet loss? (Linux)

For a general sense of the scale of your problem netstat -s will track your total number of retransmissions.

# netstat -s | grep retransmitted
     368644 segments retransmitted

You can aso grep for segments to get a more detailed view:

# netstat -s | grep segments
         149840 segments received
         150373 segments sent out
         161 segments retransmitted
         13 bad segments received

For a deeper dive, you'll probably want to fire up Wireshark.

In Wireshark set your filter to tcp.analysis.retransmission to see retransmissions by flow.

That's the best option I can come up with.

Other dead ends explored:

netfilter/conntrack tools don't seem to keep retransmits
stracing netstat -s showed that it is just printing /proc/net/netstat
column 9 in /proc/net/tcp looked promising, but it unfortunately appears to be unused.

Best Answer

Related Solutions

ISA Server dropping packets as it believes they are spoofed

Linux – How passively monitor for tcp packet loss? (Linux)

Related Topic