This is a Canonical Question about Server Security – Responding to Breach Events (Hacking)
See Also:
Canonical Version
I suspect that one or more of my servers is compromised by a hacker, virus, or other mechanism:
- What are my first steps? When I arrive on site should I disconnect the server, preserve "evidence", are there other initial considerations?
- How do I go about getting services back online?
- How do I prevent the same thing from happening immediately again?
- Are there best practices or methodologies for learning from this incident?
- If I wanted to put a Incident Response Plan together, where would I start? Should this be part of my Disaster Recovery or Business Continuity Planning?
Original Version
2011.01.02 – I'm on my way into work at 9.30 p.m. on a Sunday because our server has been compromised somehow and was resulting in a
DOS attack on our provider. The servers access to the Internet
has been shut down which means over 5-600 of our clients sites are now
down. Now this could be an FTP hack, or some weakness in code
somewhere. I'm not sure till I get there.How can I track this down quickly? We're in for a whole lot of
litigation if I don't get the server back up ASAP. Any help is
appreciated. We are running Open SUSE 11.0.
2011.01.03 – Thanks to everyone for your help. Luckily I WASN'T the only person responsible for this server, just the nearest. We managed
to resolve this problem, although it may not apply to many others in a
different situation. I'll detail what we did.We unplugged the server from the net. It was performing (attempting to
perform) a Denial Of Service attack on another server in Indonesia,
and the guilty party was also based there.We firstly tried to identify where on the server this was coming from,
considering we have over 500 sites on the server, we expected to be
moonlighting for some time. However, with SSH access still, we ran a
command to find all files edited or created in the time the attacks
started. Luckily, the offending file was created over the winter
holidays which meant that not many other files were created on the
server at that time.We were then able to identify the offending file which was inside the
uploaded images folder within a ZenCart website.After a short cigarette break we concluded that, due to the files
location, it must have been uploaded via a file upload facility that
was inadequetly secured. After some googling, we found that there was
a security vulnerability that allowed files to be uploaded, within the
ZenCart admin panel, for a picture for a record company. (The section
that it never really even used), posting this form just uploaded any
file, it did not check the extension of the file, and didn't even
check to see if the user was logged in.This meant that any files could be uploaded, including a PHP file for
the attack. We secured the vulnerability with ZenCart on the infected
site, and removed the offending files.The job was done, and I was home for 2 a.m.
The Moral
– Always apply security patches for ZenCart, or any other CMS system for that matter. As when security updates are released, the whole
world is made aware of the vulnerability.
– Always do backups, and backup your backups.
– Employ or arrange for someone that will be there in times like these. To prevent anyone from relying on a panicy post on Server
Fault.
Best Answer
It's hard to give specific advice from what you've posted here but I do have some generic advice based on a post I wrote ages ago back when I could still be bothered to blog.
Don't Panic
First things first, there are no "quick fixes" other than restoring your system from a backup taken prior to the intrusion, and this has at least two problems.
This question keeps being asked repeatedly by the victims of hackers breaking into their web server. The answers very rarely change, but people keep asking the question. I'm not sure why. Perhaps people just don't like the answers they've seen when searching for help, or they can't find someone they trust to give them advice. Or perhaps people read an answer to this question and focus too much on the 5% of why their case is special and different from the answers they can find online and miss the 95% of the question and answer where their case is near enough the same as the one they read online.
That brings me to the first important nugget of information. I really do appreciate that you are a special unique snowflake. I appreciate that your website is too, as it's a reflection of you and your business or at the very least, your hard work on behalf of an employer. But to someone on the outside looking in, whether a computer security person looking at the problem to try and help you or even the attacker himself, it is very likely that your problem will be at least 95% identical to every other case they've ever looked at.
Don't take the attack personally, and don't take the recommendations that follow here or that you get from other people personally. If you are reading this after just becoming the victim of a website hack then I really am sorry, and I really hope you can find something helpful here, but this is not the time to let your ego get in the way of what you need to do.
You have just found out that your server(s) got hacked. Now what?
Do not panic. Absolutely do not act in haste, and absolutely do not try and pretend things never happened and not act at all.
First: understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact.
Some of these steps are going to hurt, and (unless your website holds a copy of my details) I really don't care if you ignore all or some of these steps, that's up to you. But following them properly will make things better in the end. The medicine might taste awful but sometimes you have to overlook that if you really want the cure to work.
Stop the problem from becoming worse than it already is:
However annoyed your customers might be to have you tell them about a problem, they'll be far more annoyed if you don't tell them, and they only find out for themselves after someone charges $8,000 worth of goods using the credit card details they stole from your site.
Remember what I said previously? The bad thing has already happened. The only question now is how well you deal with it.
Understand the problem fully:
Why not just "repair" the exploit or rootkit you've detected and put the system back online?
In situations like this the problem is that you don't have control of that system any more. It's not your computer any more.
The only way to be certain that you've got control of the system is to rebuild the system. While there's a lot of value in finding and fixing the exploit used to break into the system, you can't be sure about what else has been done to the system once the intruders gained control (indeed, its not unheard of for hackers that recruit systems into a botnet to patch the exploits they used themselves, to safeguard "their" new computer from other hackers, as well as installing their rootkit).
Make a plan for recovery and to bring your website back online and stick to it:
Nobody wants to be offline for longer than they have to be. That's a given. If this website is a revenue generating mechanism then the pressure to bring it back online quickly will be intense. Even if the only thing at stake is your / your company's reputation, this is still going generate a lot of pressure to put things back up quickly.
However, don't give in to the temptation to go back online too quickly. Instead move with as fast as possible to understand what caused the problem and to solve it before you go back online or else you will almost certainly fall victim to an intrusion once again, and remember, "to get hacked once can be classed as misfortune; to get hacked again straight afterward looks like carelessness" (with apologies to Oscar Wilde).
Reducing the risk in the future.
The first thing you need to understand is that security is a process that you have to apply throughout the entire life-cycle of designing, deploying and maintaining an Internet-facing system, not something you can slap a few layers over your code afterwards like cheap paint. To be properly secure, a service and an application need to be designed from the start with this in mind as one of the major goals of the project. I realise that's boring and you've heard it all before and that I "just don't realise the pressure man" of getting your beta web2.0 (beta) service into beta status on the web, but the fact is that this keeps getting repeated because it was true the first time it was said and it hasn't yet become a lie.
You can't eliminate risk. You shouldn't even try to do that. What you should do however is to understand which security risks are important to you, and understand how to manage and reduce both the impact of the risk and the probability that the risk will occur.
What steps can you take to reduce the probability of an attack being successful?
For example:
What steps can you take to reduce the consequences of a successful attack?
If you decide that the "risk" of the lower floor of your home flooding is high, but not high enough to warrant moving, you should at least move the irreplaceable family heirlooms upstairs. Right?
... And finally
I've probably left out no end of stuff that others consider important, but the steps above should at least help you start sorting things out if you are unlucky enough to fall victim to hackers.
Above all: Don't panic. Think before you act. Act firmly once you've made a decision, and leave a comment below if you have something to add to my list of steps.