Tcpdump only showing output after stopping it

tcpdump

I am trying to see if I get ping packets through one interface but tcpdump shows nothing. However, as soon as I stop it (Ctrl + C), all the packets appear and the summary looks good.

For example:

11:49:45.486887 IP 11.0.0.1 > 11.0.0.9: ICMP echo reply, id 13532, seq 1, length 64

And here the summary:

10 packets captured
10 packets received by filter
0 packets dropped by kernel

I normally see the traffic "live" but I don't understand why now it is hidden until I stop tcpdump. Can somebody point me to what might be happening?

Best Answer

You probably want the -l or -U option. See the info in man page ...

   -l     Make  stdout  line buffered.  Useful if you want to see the data
          while capturing it.
          ...
          -U is similar to -l in its behavior, but it will cause output to
          be  ``packet-buffered'', so that the output is written to stdout
          at the end of each packet rather than at the end of  each  line;

Related Solutions

Linux – Clarification about Linux TCP window size and delays

After doing a little more digging into my traffic, I was able to see that my data was nothing but a sequence of small bursts with small idle periods between them.

With the useful tool ss, I was able to retrieve the current congestion window size of my connection (see the cwnd value in the output):

[user@localhost ~]$ /usr/sbin/ss -i -t -e | grep -A 1 56001

ESTAB 0 0 192.168.1.1:56001
192.168.2.1:45614 uid:1001 ino:6873875 sk:17cd4200ffff8804 ts sackscalable wscale:8,9 rto:277 rtt:74/1 ato:40 cwnd:36 send 5.6Mbps rcv_space:5792

I ran the tool several times and discovered that the congestion window size was regularly reset to the initial value (10ms, on my Linux box). The connection was constantly looping back to the slow start phase. During the slow start period, bursts with a number of messages exceeding the window size were delayed, waiting for the acks related to the first packets of the burst.

The fact that the traffic consists of a sequence of bursts likely explains the reset of the congestion window size.

By deactivating the slow start mode after idle period, I was able to get rid of the delays.

[user@host ~]$ cat /proc/sys/net/ipv4/tcp_slow_start_after_idle 0

Log tcpdump Output

Instead of logging all traffic, I would suggest the following: Monitor the number of packets sent to your server. If it exceeds a certain threshold, log a couple of 1000 packets, then wait for a longer time.

That packet trace should contain plenty of information which can be used for analysis. Also, it will not impose too much additional load on your server while everything is fine. You could use the following hacked together bash code as a starting point (could be started in screen, for example):

interface=eth0
dumpdir=/tmp/

while /bin/true; do
  pkt_old=`grep $interface: /proc/net/dev | cut -d :  -f2 | awk '{ print $2 }'`
  sleep 1
  pkt_new=`grep $interface: /proc/net/dev | cut -d :  -f2 | awk '{ print $2 }'`

  pkt=$(( $pkt_new - $pkt_old ))
  echo -ne "\r$pkt packets/s\033[0K"

  if [ $pkt -gt 5000 ]; then
    echo -e "\n`date` Under attack, dumping packets."
    tcpdump -n -s0 -c 2000 -w $dumpdir/dump.`date +"%Y%m%d-%H%M%S"`.cap
    echo "`date` Packets dumped, sleeping now."
    sleep 300
  fi
done

Feel free to adapt it to your needs.

Best Answer

Related Solutions

Linux – Clarification about Linux TCP window size and delays

Log tcpdump Output

Related Topic