Apache – Identifying Spam Sources on CentOS/Apache/suPHP Server

apache-2.2eximPHPsuphp

My server is sending lots of spam and I searched for the problem for hours now. After googling I found a forum where they talked about this and mentioned to dig into the exim log, so I did and found that emails were sent from: [username]@vps1.[hostname].[tld]. In the forum they said the e-mails were probably sent from my server because this is not a used e-mail address. They also mentioned to dig into php logs.

I've tried this but couldn't find anything, so via e-mail headers I'm now trying to detect the script sending all these e-mails. Which is were I'm stuck now.

I've changed php.ini by adding the following rules:

mail.add_x_header = On
mail.log = /var/log/phpmail.log

Also I've added exim.conf by adding this line:

+arguments \

Restarted exim and apache, but I don't see any X-PHP-Script headers in the exim log and the php mail log isn't created.

Only thing I see is an X header in the exim log:

X=TLSv1:RC4-SHA:128

Can anyone tell me what to do next?

EDIT

Here are some lines from the exim log:

bash-3.2# cat /var/log/exim/mainlog | grep 1W9FsC-0003qq-S2
2014-01-31 16:19:16 1W9FsC-0003qq-S2 <= instijl@vps1.xxx.nl U=instijl P=local S=816 T="Re:  It's good to see you," from <instijl@vps1.xxx.nl> for richisone@bigpond.com
2014-01-31 16:19:16 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9FsC-0003qq-S2
2014-01-31 16:19:17 1W9FsC-0003qq-S2 ** richisone@bigpond.com F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp: SMTP error from remote mail server after initial connection: host extmail.bigpond.com [61.9.168.122]: 554 nskntcmgw02p BigPond Inbound IB103. Connection refused. 141.138.199.65 has a poor reputation on the Cloudmark Sender Intelligence (CSI) list. Please visit http://csi.cloudmark.com/reset-request/?ip=141.138.199.65 to request a delisting.
2014-01-31 16:19:17 cwd=/var/spool/exim 7 args: /usr/sbin/exim -t -oem -oi -f <> -E1W9FsC-0003qq-S2
2014-01-31 16:19:17 1W9FsD-0003r9-H9 <= <> R=1W9FsC-0003qq-S2 U=mail P=local S=2006 T="Mail delivery failed: returning message to sender" from <> for instijl@vps1.xxx.nl
2014-01-31 16:19:17 1W9FsC-0003qq-S2 Completed

bash-3.2# cat /var/log/exim/mainlog | grep 1W9FsC-0003qc-M7
2014-01-31 16:19:16 1W9FsC-0003qc-M7 <= instijl@vps1.xxx.nl U=instijl P=local S=822 T="Re:  It's good to see you," from <instijl@vps1.xxx.nl> for richisingh7710@gmail.com
2014-01-31 16:19:16 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9FsC-0003qc-M7
2014-01-31 16:19:17 1W9FsC-0003qc-M7 ** richisingh7710@gmail.com F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp: SMTP error from remote mail server after end of data: host gmail-smtp-in.l.google.com [173.194.65.26]: 550-5.7.1 [141.138.199.65      12] Our system has detected that this message is\n550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,\n550-5.7.1 this message has been blocked. Please visit\n550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for\n550 5.7.1 more information. y48si18631040eew.58 - gsmtp
2014-01-31 16:19:17 cwd=/var/spool/exim 7 args: /usr/sbin/exim -t -oem -oi -f <> -E1W9FsC-0003qc-M7
2014-01-31 16:19:17 1W9FsD-0003r1-BS <= <> R=1W9FsC-0003qc-M7 U=mail P=local S=2146 T="Mail delivery failed: returning message to sender" from <> for instijl@vps1.xxx.nl
2014-01-31 16:19:17 1W9FsC-0003qc-M7 Completed

bash-3.2# cat /var/log/exim/mainlog | grep 1W9Frw-0003oS-Gd
2014-01-31 16:19:00 1W9Frw-0003oS-Gd <= instijl@vps1.xxx.nl U=instijl P=local S=823 T="FW:  Yo" from <instijl@vps1.xxx.nl> for ketabatgooll@yahoo.com
2014-01-31 16:19:00 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9Frw-0003oS-Gd
2014-01-31 16:19:02 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.136.217.203]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:03 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.136.216.26]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:04 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.36]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:06 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.138.112.33]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:07 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.35]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:07 1W9Frw-0003oS-Gd == ketabatgooll@yahoo.com R=lookuphost T=remote_smtp defer (-45): SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.35]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

bash-3.2# cat /var/log/exim/mainlog | grep 1W9Frg-0003mP-S6
2014-01-31 16:18:44 1W9Frg-0003mP-S6 <= instijl@vps1.xxx.nl U=instijl P=local S=814 T="call me" from <instijl@vps1.xxx.nl> for ket@web.de
2014-01-31 16:18:44 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9Frg-0003mP-S6
2014-01-31 16:18:45 1W9Frg-0003mP-S6 => ket@web.de F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp S=837 H=mx-ha03.web.de [213.165.67.104] X=TLSv1:AES256-SHA:256 C="250 Requested mail action okay, completed: id=0Le6s0-1VUM4v1jno-00pvEX"
2014-01-31 16:18:45 1W9Frg-0003mP-S6 Completed

Best Answer

Summary of TroubleShooting steps

The "U=instijl" shown from your /var/log/exim/mainlog excerpt tells you that whatever is sending the emails is running as user instijl. First see if the user is logged in with a shell. Second use 'ps aux' to find if any processes are running by that user. Third, look in your apache access logs to see what traffic is being sent to apache at the exact same time as 4 mails above. I suspect you have an insecure "send me feedback" form that is being abused (insecure because you allow incoming http request to set the sender, the recipient, and the message body).

If the virtual host that is serving and accepting this request doesn't have its own access log entry, it won't log to the general access log (which is likely what you found). Find the specific section which is answering requests for that user and add the access log entry (or if it's already logging, figure out the filename). If you run 'httpd -S', apache prints out basic virtual host configuration to help you more easily find where in the config file that section is controlled/configured.

Another thing you can do is 'yum install ngrep' (may be in external repo such as epel) and run 'ngrep -n -q port 80' and see what traffic is coming in. A more specific command which only shows incoming requests would be "ngrep -q -s 240 'GET|POST' port 80". Adjust the 240 up or down if you want to see more or less of the request, or omit it if you want to see the full request.

Related Topic