TCPDUMP – How to Calculate Packet Loss from a Binary TCPDUMP File

we found the root cause of this. We had an acceptCount of 25 in our tomcat server.xml.

acceptCount is documented like this:

acceptCount

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 100.

But this is not the whole story about acceptCount. Short: acceptCount is the backlog Parameter when opening the socket. So this value is important for the listen backlog, even if not all threads are busy. It is important if request are faster coming in then tomcat can accept and delegate them to waiting threads. The default acceptCount is 100. This is still a small value to feed a sudden peak in requests.

We checked the same thing with apache and nginx and had the same strange packet loss but with higher concurrency values. The corresponding value in apache is ListenBacklog which defaults to 511.

BUT, with debian (and other linux based os) the default max value for the backlog paramter is 128.

$ sysctl -a | grep somaxc
net.core.somaxconn = 128

So whatever you type in acceptCount or ListenBacklog it will not be over 128 until you change net.core.somaxconn

For a very busy webserver 128 is not enough. You should change it to something like 500, 1000 or 3000, depending on your needs.

After setting acceptCount to 1000 and net.core.somaxconn to 1000 we no longer had those dropped packets. (Now we have a bottleneck somewhere else, but this is another story..)

My best bet would be to use something like:

 tcpdump -ieth0 -s96 -w traffic.dump 'ip or icmp or tcp or udp'

Where the "tricky" part will be to chose a correct value for the "-s" (snaplen) parameter (snaplen is the maximum length of the packet tcpdump will capture).

From the tcpdump man pages:

Snarf snaplen bytes of data from each packet rather than the default of 68 (with NIT, the minimum is actually 96). 68 bytes is adequate for IP, ICMP, TCP and UDP but may truncate protocol information from name server and NFS packets (see below). Packets truncated because of a limited snapshot are indicated in the output with ``[|proto]'', where proto is the name of the protocol level at which the truncation has occurred. Note that taking larger snapshots both increases the amount of time it takes to process packets and, effectively, decreases the amount of packet buffering. This may cause packets to be lost. You should limit snaplen to the smallest number that will capture the protocol information you're interested in.

In this example i'm using 96 to be "almost" sure that I would capture 100% of ethernet+ip+(icmp || udp || tcp) header values.

In case your traffic have IP or TCP options (i.e. timestamps) and you want to also capture this info, then you will have to play with the snaplen parameter (i.e. increase/decrease it).

In case the length of the headers of your packet is less than snaplen, you may also capture part of the payload.

Finally, to read the traffic captured, I would use something like:

tcpdump -e -nn -vv -r traffic.dump   

Where the important part is to use the "-e" option so you can get the ethernet headers printed.

This page gives you an idea about the size of the ethernet/tcp/udp headers under different circumstances and may help you to arrive to a "correct" value for the snaplen parameter.