Linux – Hacked server with IPmech installed in /var/tmp/.aw running cronjob every minute, how to determine how I was hacked

hackinglinuxUbuntu

One of my user accounts on an Ubuntu 10.04.3 server was hacked, and I'm not sure how. The password was strong. A cronjob was installed in my user's crontab running and executable in /var/tmp/.aw

The /var/tmp/.aw directory contained a collection of executables including one called bash.

I've examined my ~/.bash_history and found some very suspicious stuff. I provide the relevant snippets below.

    w
    ls
    passwd
    cd /var/tmp
    w
    ls
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    w
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    w
    cat /prooc/cpuinfo
    cat /proc/cpuinfo
    exotr
    wq
    w
    exit
    w
    ls
    passwd
    cd /var/tmp
    ls
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    ls
    tar xzvf IPmech.tgz
    rm -rf IPmech.tgz
    cd .aw
    s
    ls
    ./autorun
    chmod +x *
    ./autorun
    ./start TKLL
    ls
    rm -rf m.ses
    ps x
    kill -9 4350
    ls
    ps x
    rm -rf m.ses
    kill -9  4460
    ls
    ps x
    w
    ls
    nano 192.168.0.100.user2
    rm -rf *seeN8
    ls
    rm -rf *see*
    ls
    nano m.set
    rm -rf m.ses
    ps x
    kill -9 4582
    ls
    ps x
    kill -9 4645
    rm -rf m.ses
    ls
    ps x
    kill -9 4693
    ls
    rm -rf m.ses
    ps x
    kill -9 4733
    rm -rf m.ses
    ps x
    kill -9 4757
    ls
    nano m.set
    rm -rf m.ses
    ps x
    kill -9 4800
    we
    w
    ls
    ps x
    kill -9 4878
    ls
    rm -rf m.ses
    ps x
    kill -9 4926
    ls
    w
    ps x
    ls
    kill -9 4964
    w
    exit
    w
    ls
    ps x
    cd /var/tmp
    w
    ls
    exit
    sudo su
    passwd
    ls
    ls -al
    ls .ssh/
    rm id_dsa.pub 
    touch .sudo_as_admin_successful 
    sudo su
    passwd
    it is
    sudo su
    w
    echo "yay :D" > /dev/pts/9
    echo "I take it it's working..." > /dev/pts/9
    w
    echo "Is this annoying???" > /dev/pts/9
    w
    exit

Specific questions:

  1. What is exotr? I can't locate or find it with which, nor is there man entry for it
  2. What is IPmech, a google search yields a bunch of stuff discussing ceramics and modeling cracks in ceramics. IPMech seems to be the Institute for problems in mechanics, part of the russian academy of sciences. I'm not sure ho legitimate that is though. Also I still don't know how they got into the account in the first place. The logs don't go far enough back to see the log in attempt that corresponds to the bask history.
  3. I have removed the crontab completely (there was nothing else in it) and deleted /var/tmp/.aw, and rebooted the server. I've checked the running processes for anything funny and everything looks legit. I've changed my password. Do I need to change my public keys as well?
  4. What else can/should I look for to help identify the intrusion mechanism?

Thanks

Best Answer

I wrote an answer over on Security.SE a while ago with techniques for finding out how they got in. It's not a comprehensive answer as such a thing would fill an entire book.

The gist of it is: "Look in the logs; identify suspicious things (files, log entries) by timestamps".

In your case, use whatever logs you have to find his IP address (last -i or grep username /var/log/auth.log) and then look through all other logs (particularly web server logs if you run one) for that IP address. Searching all your logs for IPmech may also be useful. If you can find where he got it from you might be able to get a copy yourself and see what it does. My guess (based on my own Google search for IPmech) is that it was running an open proxy.

exotr looks like a typo of exit to me. He hit "o" instead of "i" and mashed "tr" instead of just "t". Especially since he successfully typed "exit" two commands later.

Cleaning up the server is all well and good but even if "everything looks legit", you can never be sure. Wiping and re-installing using a backup from before the compromise is a way you can be sure. See the copious advice in My server's been hacked EMERGENCY. It's good practice to do this anyway as backups aren't really backups until you have successfully restored from them.

Related Topic