Linux – Pain removing a perl rootkit

linuxperlprocessrootkitSecurity

So, we host a geoservice webserver thing at the office.

Someone apparently broke into this box (probably via ftp or ssh), and put some kind of irc-managed rootkit thing.

Now I'm trying to clean the whole thing up, I found the process pid who tries to connect via irc, but i can't figure out who's the invoking process (already looked with ps, pstree, lsof)
The process is a perl script owned by www user, but ps aux |grep displays a fake file path on the last column.

Is there another way to trace that pid and catch the invoker?

Forgot to mention: the kernel is 2.6.23, which is exploitable to become root, but I can't touch this machine too much, so I can't upgrade the kernel

EDIT: lsof might help:

lsof -p 9481
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEss
perl 9481 www cwd DIR 8,2 608 2 /ss
perl 9481 www rtd DIR 8,2 608 2 /ss
perl 9481 www txt REG 8,2 1168928 38385 /usr/bin/perl5.8.8ss
perl 9481 www mem REG 8,2 135348 23286 /lib64/ld-2.5.soss
perl 9481 www mem REG 8,2 103711 23295 /lib64/libnsl-2.5.soss
perl 9481 www mem REG 8,2 19112 23292 /lib64/libdl-2.5.soss
perl 9481 www mem REG 8,2 586243 23293 /lib64/libm-2.5.soss
perl 9481 www mem REG 8,2 27041 23291 /lib64/libcrypt-2.5.soss
perl 9481 www mem REG 8,2 14262 23307 /lib64/libutil-2.5.soss
perl 9481 www mem REG 8,2 128642 23303 /lib64/libpthread-2.5.soss
perl 9481 www mem REG 8,2 1602809 23289 /lib64/libc-2.5.soss
perl 9481 www mem REG 8,2 19256 38662 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/IO/IO.soss
perl 9481 www mem REG 8,2 21328 38877 /usr/lib64/perl5/5.8.8/x86_64-linux-threa d-multi/auto/Socket/Socket.soss
perl 9481 www mem REG 8,2 52512 23298 /lib64/libnss_files-2.5.soss
perl 9481 www 0r FIFO 0,5 1068892 pipess
perl 9481 www 1w FIFO 0,5 1071920 pipess
perl 9481 www 2w FIFO 0,5 1068894 pipess
perl 9481 www 3u IPv4 130646198 TCP 192.168.90.7:60321->www.****.net:ircd (SYN_SENT)

Best Answer

If I can give you any advice, it is to stop wasting your time cleaning up. Make an image of the OS for forensic stuff for later, and just reinstall the server.

Sorry, but its the only secure way to resolving yourself from being rootkitted.

Later you can check the image, for certain reasons, why it happened.

From my own personal experience, I did this, and later found an internal user which had a SSH key containing the flaw of openssl in 2008.

I hope, it clears up the things.

Note:
If you are going to image/backup the server before reinstalling, be very careful, how you do this. As @dfranke said, boot from a trusted medium to backup.

You shouldn't connect to other machines from a rooted server, as great rootkits are known to be able to spread through trusted sessions such as SSH.