As of GNU coreutils 7.5 released in August 2009, sort
allows a -h
parameter, which allows numeric suffixes of the kind produced by du -h
:
du -hs * | sort -h
If you are using a sort that does not support -h
, you can install GNU Coreutils. E.g. on an older Mac OS X:
brew install coreutils
du -hs * | gsort -h
From sort
manual:
-h, --human-numeric-sort compare human readable numbers (e.g., 2K 1G)
Linux auditing can help. It will at least locate users and processes making datagram network connections. UDP packets are datagrams.
First, install the auditd
framework on your platform and ensure that auditctl -l
returns something, even if it says that no rules are defined.
Then, add a rule to watch the system call socket()
and tag it for easy finding later (-k
). I need to assume that you are on a 64-bit architecture, but you can substitute b32
in place of the b64
if you aren't.
auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
You have to pick through man pages and header files to build this, but what it captures is essentially this system call: socket(PF_INET, SOCK_DGRAM|X, Y)
, where the third parameter is unspecified but frequently zero. PF_INET
is 2 and SOCK_DGRAM
is 2. TCP connections would use SOCK_STREAM
which would set a1=1
. (SOCK_DGRAM
in the second parameter may be ORed with SOCK_NONBLOCK
or SOCK_CLOEXEC
, hence the &=
comparison.) The -k SOCKET
is our keyword we want to use when searching audit trails later. It can be anything, but I like to keep it simple.
Let a few moments go by and review the audit trails. Optionally, you could force a couple of packets by pinging a host out on the net, which will cause a DNS lookup to occur, which uses UDP, which should trip our audit alert.
ausearch -i -ts today -k SOCKET
And output similar to the section below will appear. I'm abbreviating it to highlight the important parts
type=SYSCALL ... arch=x86_64 syscall=socket success=yes exit=1 a0=2 a1=2 ... pid=14510 ... auid=zlagtime uid=zlagtime ... euid=zlagtime ... comm=ping exe=/usr/bin/ping key=SOCKET
In the above output, we can see that the ping
command caused the socket to be opened. I could then run strace -p 14510
on the process, if it was still running. The ppid
(parent process ID) is also listed in case it is a script that spawns the problem child a lot.
Now, if you have a lot of UDP traffic, this isn't going to be good enough and you'll have to resort to OProfile or SystemTap, both of which are currently beyond my expertise.
This should help narrow things down in the general case.
When you are done, remove the audit rule by using the same line you used to create it, only substitute -a
with -d
.
auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET
Best Answer
tcpdump isn't guaranteed to process all packets. There's some buffering, but if the rate of packets crossing the network interface is faster than the rate at which your CPU can run them through tcpdump, the kernel starts dropping packets. The higher the demand on the CPU, and the higher the network traffic rate, the higher the propensity for dropping (it's impossible to be specific, you'll have to test this on your systems to find out where the drop threshold is).
Offhand, I don't know.
As for better ways, the term you want is "traffic accounting". This is built into IPTables, so any modern Linux distro should support it out-of-the-box. In short, a few simple "pass-through" IPTables rules can give you the total bytes transfered, in real time, for just about any specified traffic types (broken down by proto, port, IP, etc., or not) that you want.
There's a great walk-through, with specific commands, here: http://www.catonmat.net/blog/traffic-accounting-with-iptables
This should be much more lightweight and reliable than tcpdump, since Netfilter handles it entirely in the kernel, and the kernel has the packet length info, anyway.