Linux – How did you implement log management on your servers

linuxlog-filesscriptingsyslog

I'm trying to figure out how other people implement their log management systems.

I have 20-30 Linux servers and a few Windows boxes (most of them virtualized). We utilize a lot of Perl and Bash scripts to do most of our automated jobs and I'm trying to standardize their logging.

I've been looking at log4perl and log4sh for logging of scripts and syslog-ng to get all the logs on a centralized logging server. I've also read up on splunk, even though is sounds like the enterprise edition is pretty pricey and I might go over the free license limit with all my servers.

I've seen other tools like swatch and logcheck, but I'm not quite sure how all these pieces fit together… Any recommendations would be greatly appreciated!

Best Answer

I've got about 30 servers, and I just use straight up syslog to send all the logs to a single logging server. For backup, all of the machines are also configured to store their own logs locally for a few days, using logrotate to take care of the rotation and deletion of old logs.

Each of my application servers runs a small perl script to send their logs to syslog, which then forwards on to the loghost (perl script below).

Then on the loghost we have some custom scripts that are similar to logcheck that basically watch the incoming logs for anything suspicious.

We also have all of the email from every host going to one place, so that if any program complains that way, we get all the messages. This could theoretically go to a single mailbox that a program could act on and analyze.

Here is my logging perl script. It works by piping the program's output into it, and then it syslogs the output and spits it back out so you can send it elsewhere (I send to multilog). You can also give it the -q option to just go to syslog.

#!/usr/bin/perl

use Sys::Syslog;
use Getopt::Long;

$SERVER_NAME = `hostname`;
chomp $SERVER_NAME;
$FACILITY = 'local0';
$PRIORITY = 'info';

GetOptions ('s=s' => \$SERVER_NAME, 'f=s' => \$FACILITY, 'p=s' => \$PRIORITY, 'q+' => \$quiet);

#print "$SERVER_NAME\n$FACILITY\n$PRIORITY\n";

#Sys::Syslog::setlogsock('unix');
openlog ($SERVER_NAME,'ndelay',$FACILITY);

if (!($quiet)) {syslog($PRIORITY,"Logging Started -- Logger version 1.1");}

$| = 1;

while (<>) {
    if (!($quiet)) {print $_ unless $_ =~ /^\s+$/};
    chomp;
    syslog($PRIORITY,$_) if $_;
}

closelog;

$| = 0;
Related Topic