I run a few Rightscale CentOS AMI based instances on Amazon EC2. Two months back I found that our SSHD security is compromised( I had added host.allow and host.deny for ssh). So I created new instances and done an IP based ssh that allows only our IPs through AWS Firewall(ec2-authorize) and chnaged the ssh 22 default port to some other port but two days back I found I was not able to login to the server and when I tried on 22 port the ssh got connected and I found that sshd_conf was changed and when I tried to edit sshd_config I found root had no write permission on the file. So I tried a chmod and it said access denied for 'root' user. This is very strange. I checked secure log and history and found nothing informative. I have PHP, Ruby On Rails, Java, WordPress apps running on these server.
This time I did a chkrootkit scan and found nothing. I renamed the /etc/ssh folder and reinstalled openssh through yum. I had faced this on 3 instances on CentOS(5.2, 5.4) I have instances on Debian as well those working fine. Is this a CentOS/Rightscale issue. Guys, what security measures I should take to prevent this.
I doubt this can be an internal hacker too since we allow only our IPs to ssh.
So I am planning to user a keylogger which can log all keystrokes from each user and terminals. Could you suggest me some Linux keylogget best suits for this purpose?
I am sure it's and ssh attack, but no idea how they did. CentOS 5 still uses openssh 4.3 so I compiled openssh 5.5 and found that they have changed the attributes for /usr/bin/ssh and /usr/sbin/sshd. I had to use lsattr filename the chattr -ia filename to remove the attributes and successfully compiled openssh. Now I am trying to setup chroot and setup Osiris for precaution.
Please share your thoughts. Also I need a keylogger logs all tty keystrokes which one you suggest?
Thanks for your support.
Best Answer
Basically, you're screwed.
If you were actually exploited, they could have added themselves a backdoor into the SSHD binary, or any other file on the system...Since the source is available, this is easy to do, and it's extremely difficult to spot if you didn't plan ahead.
If you didn't have Tripwire or similar installed before the problem happened, you're basically going to have to check every single file on the system against a known-clean install.
Easier to just move your app to a clean server.
The best prevention is a file integrity scanner. Basically, it checks every file on the system, and makes a hash of it. Then it periodically re-scans, and notifies you of any changes. Makes clean up a breeze.
I agree that you probably didn't get exploited through SSH...I've almost never seen that happen, and when it does happen, it's usually because of an unpatched vulnerability in the daemon. Brute forcing a SSH password is usually a lost cause.
Edit: I've been reading up on Osiris this afternoon, and it looks pretty cool. It periodically re-scans and sends you an email of the changed files. The enterprise version of Tripwire offers real-time checking, but of course you have to pay for it. There is an open source version of Tripwire as well, which has picked up the reins from the much much older open source tripwire, and is pretty solid as well.