Linux – awstats parse of postfix mail log drops all records

The server appears to be doing throttling which might affect you if you are testing from one IP:

2009-08-10 07:21:05 H=(aurl.domain.ni) [127.0.0.1] Warning: Sender rate 4.6 / 1h

Often issues like this are related to poor DNS. Exim does a lot of resolution during processing mail. Check the DNS config on the Exim server. Check the performance on the DNS server it is using.

Is the server doing any DNSBL processing? I'm often asked to look at servers exhibiting poor performance and it turns out they are using DNSBLs that have been retired, meaning lookups are timing out.

Try running exim in debug on the server to see where the problem is. The following will run an Exim daemon on port 26, staying attached to the console and printing lots of debug:

exim -bd -d -oX 26

Then telnet to tcp/26 instead of tcp/25 and test as you had been.

What you probably want to do here is to analyze all these logfiles once, then keep analyzing only the current logfiles from then on.

The simplest thing to do is unzip all those files into a single file, then have awstats run over it once, then point awstats at your access.log file from then on.

awstats normally has a script called logresolvemerge.pl, which can read the compressed files, and will merge them appropriately for awstats to do analsyis.

To merge all your existing ones, run

perl /usr/share/awstats/tools/logresolvemerge.pl /var/log/nginx/access.log* > /tmp/nginx.tmplog

This will probably take a while.

You can then have awstats run once over this file (set LogFile appropriately).

From then on, you should have awstats run over the most recent logfile - which is what your current configuration is doing.

Depending on how often you are running awstats vs rotating nginx logfiles, you may want to have it read both the current logfile and the previous one. (eg, if you rotate nginx logfiles every day at 12, but have awstats run every day at 1, then whenever awstats runs the logfile will only contain what's been written since the last rotation). You can use logresolvemerge.pl inside your LogFile command like this:

LogFile="/usr/share/awstats/tools/logresolvemerge.pl /var/log/nginx/access.log /var/log/nginx/access.log.1.gz |"

This tells awstats to run the logresolvemerge.pl command with the two logfiles as parameters, and awstats will read in the output of that script (that's what the pipe | does)