Running tcpdump
on local connections to an apache server, I found TCP connections being established and closed immediately every 2 seconds. How do I find which process is responsible for these? netstat -ctp
did not help, the connections were too fast and the process identifier is not displayed for TIME_WAIT ones.
They turned out to be haproxy probes, which I could verify with strace
, but I still do not know any way to pinpoint haproxy in the first place.
Best Answer
You can use the auditd framework for these kind of things. They're not very "user friendly" or intuitive, so requires a little bit of digging around on your part.
First make sure you have auditd installed, running and that your kernel supports it.
For Ubuntu you can install it with
apt-get install auditd
for example.Then you add a policy for audit to monitor all
connect
syscalls like this:If you are using a 32-bit installation of Linux you have to change b64 to b32.
This command will insert a policy to the audit framework, and any connect() syscalls will now be logged to your audit logfiles (usually
/var/log/audit/audit.log
) for you to look at.For example, a connection with netcat to news.ycombinator.com port 80 will result in something like this:
Here you can see that the /bin/nc.openbsd application initiated a connect() call, if you get lots of connect calls and only want to grep out a certain ip or port you have to do some conversion. The SOCKADDR line contains a saddr argument, it begins with 0200 followed by the port number in hexadecimal (0050) which means 80, and then the IP in hex (AE84E16A) which is news.ycombinator.com's IP of 174.132.225.106.
The audit framework can generate a lot of logs, so remember to disable it when you've accomplished your mission. To disable the above policy, simply replace -a with -d as such:
Good documentation on the auditd framework:
http://doc.opensuse.org/products/draft/SLES/SLES-security_sd_draft/part.audit.html
Convert IP adresses to/from hex, dec, binary, etc at:
http://www.kloth.net/services/iplocate.php
General hex/dec converter:
http://www.statman.info/conversions/hexadecimal.html
A Brief Introduction to auditd, from the IT Security Stack Exchange. http://security.blogoverflow.com/2013/01/a-brief-introduction-to-auditd/
Edit 1:
Another quick'n'dirty (swedish: fulhack) way to do it is to create a fast loop that dumps the connection data to you, like this:
This command uses the
ss
command (socket statistics) to dump current established connections to port 80 including what process initiated it. If its a lot of data you can add| tee /tmp/output
after done to both show the output on the screen aswell as write it to /tmp/output for later processing/digging. If it doesn't catch the quick haproxy connection, please try removingsleep 1
but be cautious of extensive logging if its a heavily utilized machine. Modify as needed!