Problem
I think my server is being used to send spam with sendmail, I'm getting a lot of mail being queued up that I don't recognize and my mail.log and syslog are getting huge.
I've shutdown sendmail, so none of it is getting out but I can't work out where it's coming from.
Investigation so far:
I've tried the solution in the blog post below and also shown in this thread.
It's meant to add a header from wherever the mail is being added and log all all mail to file, so I changed the following lines in my php.ini file:
mail.add_x_header = On
mail.log = /var/log/phpmail.log
But nothing is appearing in the phpmail.log.
I used the command here to investigate cron jobs for all users, but nothing is out of place. The only cron being run is the cron for the website.
And then I brought up all php files which had been modified in the last 30 days but none of them look suspicious.
What else can I do to find where this is coming from?
Mail.log reports
Turned sendmail back on for second. Here is a small sample of the reports:
Jun 10 14:40:30 ubuntu12 sm-mta[13684]: s5ADeQdp013684: from=<>, size=2431, class=0, nrcpts=1, msgid=<201406101220.s5ACK1cC011438@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:30 ubuntu12 sm-msp-queue[13674]: s5ACK1cC011438: to=www-data, delay=01:20:14, xdelay=00:00:00, mailer=relay, pri=571670, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdp013684 Message accepted for delivery)
Jun 10 14:40:30 ubuntu12 sm-mta[13719]: s5ADeQdp013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32683, dsn=2.0.0, stat=Sent
Jun 10 14:40:30 ubuntu12 sm-mta[13684]: s5ADeQdr013684: from=<www-data@ubuntu12.pcsmarthosting.co.uk>, size=677, class=0, nrcpts=1, msgid=<201406101200.s5AC0gpi011125@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:31 ubuntu12 sm-msp-queue[13674]: s5AC0gpi011125: to=www-data, ctladdr=www-data (33/33), delay=01:39:49, xdelay=00:00:01, mailer=relay, pri=660349, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdr013684 Message accepted for delivery)
Jun 10 14:40:31 ubuntu12 sm-mta[13721]: s5ADeQdr013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, ctladdr=<www-data@ubuntu12.pcsmarthosting.co.uk> (33/33), delay=00:00:01, xdelay=00:00:00, mailer=local, pri=30946, dsn=2.0.0, stat=Sent
Jun 10 14:40:31 ubuntu12 sm-mta[13684]: s5ADeQdt013684: from=<www-data@ubuntu12.pcsmarthosting.co.uk>, size=677, class=0, nrcpts=1, msgid=<201406101215.s5ACF2Nq011240@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:31 ubuntu12 sm-msp-queue[13674]: s5ACF2Nq011240: to=www-data, ctladdr=www-data (33/33), delay=01:25:29, xdelay=00:00:00, mailer=relay, pri=660349, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdt013684 Message accepted for delivery)
Jun 10 14:40:31 ubuntu12 sm-mta[13723]: s5ADeQdt013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, ctladdr=<www-data@ubuntu12.pcsmarthosting.co.uk> (33/33), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30946, dsn=2.0.0, stat=Sent
Ju
Further Investigation
Spotted 4 spam accounts registered in the past day, which is suspicious however all have normal user privileges.
There are no contact forms on the site, there are a number of forms and they take either filtered text input or plain text input.
Mail is still being queued up having switched the website to maintenance mode, which blocks out everyone but the admin.
Ok more investigation, it looks like the email is being send by my websites cron which runs every 5 minutes. However there are no cron jobs I've set-up which run more than once an hour and show on the website log so presumably someone has managed to edit my cron somehow.
Problem Over:
Turns out most of this was ignorance on my part. Cron tries to send an email when it runs. Because the cron was run by www-data, it tried to send it to www-data. The peculiar address was because I had never changed my dnshostname from the server default, which for some weird reason was pcsmarthosting.co.uk. (Weird because it's not at all related to my host.)
As I found out the format of the default address of www-data is hostname@dnshostname.
Copy of email:
V8
T1402410301
K1402411201
N2
P120349
I253/1/369045
MDeferred: Connection refused by [127.0.0.1]
Fbs
$_www-data@localhost
${daemon_flags}c u
Swww-data
Awww-data@ubuntu12.pcsmarthosting.co.uk
MDeferred: Connection refused by [127.0.0.1]
C:www-data
rRFC822; www-data@ubuntu12.pcsmarthosting.co.uk
RPFD:www-data
H?P?Return-Path: <�g>
H??Received: (from www-data@localhost)
by ubuntu12.pcsmarthosting.co.uk (8.14.4/8.14.4/Submit) id s5AEP13T015507
for www-data; Tue, 10 Jun 2014 15:25:01 +0100
H?D?Date: Tue, 10 Jun 2014 15:25:01 +0100
H?x?Full-Name: CronDaemon
H?M?Message-Id: <201406101425.s5AEP13T015507@ubuntu12.pcsmarthosting.co.uk>
H??From: root (Cron Daemon)
H??To: www-data
H??Subject: Cron <www-data@ubuntu12> /usr/bin/drush @main elysia-cron
H??Content-Type: text/plain; charset=ANSI_X3.4-1968
H??X-Cron-Env: <PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin>
H??X-Cron-Env: <COLUMNS=80>
H??X-Cron-Env: <SHELL=/bin/sh>
H??X-Cron-Env: <HOME=/var/www>
H??X-Cron-Env: <LOGNAME=www-data>
Best Answer
As noted by Alien Life Form, The mail looks to have originated from the www-data user. You haven't given enough info to conclude whether or not the problem is a "poorly coded contact form".
It may well be that some web application that is either being abused, or is misconfigured and is trying to tell you. The content of the emails should make that clear. Look at a few files in your mail spool if it's not otherwise easy to get a copy.
The very high frequency of these entries is notable, as is the fact that www-data seems to be mailing itself. Is this a mailing loop? That's particularly likely if you're not otherwise dealing with multiple php requests per second. Also www-data doesn't seem like the most likely target for spam.
Make sure that the www-data account is aliased to something sensible. eg you might use /etc/aliases (and run newaliases after editing that file). Depending on your distribution, that might be in a different location like /etc/mail/aliases.
If the problem is not a mail loop, then you presumably have web requests coming in at high speed associated with the emails being generated. You could capture some web traffic with tcpdump, and look through it (eg maybe with wireshark) to find the web requests involved (eg look for some content from the emails at the time). Unless it's coming via https, you'll then have the originating IP, and the URL.
You can likely also get the IP and URL from your http access logs, based on the timing of requests. How many URLs in there are accessed many times per second? Unless you have a very high traffic server, that's likely easier than capturing traffic. It also might allow you to quickly rule out web accesses being the source of the emails.
--
I see you've commented on Alien Life Form's answer that mail is being queued up 'every 5 minutes or so', which is a lot less frequent than in the log you provided. Given that, I'd be looking to line up the timing of those mail submissions with hits on the same URL in your access log (particularly look at POST requests, as they're more likely).
Are the requests exactly every 5 minutes? Do you have a 5 minute cron job?
Knowing a bit about the content of the emails would help quite a bit here in knowing which things to look for first.