I'm running several RHEL based systems which utilize the audit functionality within the 2.6 kernel to track user activity and I need to have these logs sent to centralized SYSLOG servers for monitoring and event correlation. Anyone know how to achieve this?
Linux – Sending audit logs to SYSLOG server
auditlinuxredhatsyslog
Related Solutions
I've got about 30 servers, and I just use straight up syslog to send all the logs to a single logging server. For backup, all of the machines are also configured to store their own logs locally for a few days, using logrotate to take care of the rotation and deletion of old logs.
Each of my application servers runs a small perl script to send their logs to syslog, which then forwards on to the loghost (perl script below).
Then on the loghost we have some custom scripts that are similar to logcheck that basically watch the incoming logs for anything suspicious.
We also have all of the email from every host going to one place, so that if any program complains that way, we get all the messages. This could theoretically go to a single mailbox that a program could act on and analyze.
Here is my logging perl script. It works by piping the program's output into it, and then it syslogs the output and spits it back out so you can send it elsewhere (I send to multilog). You can also give it the -q option to just go to syslog.
#!/usr/bin/perl
use Sys::Syslog;
use Getopt::Long;
$SERVER_NAME = `hostname`;
chomp $SERVER_NAME;
$FACILITY = 'local0';
$PRIORITY = 'info';
GetOptions ('s=s' => \$SERVER_NAME, 'f=s' => \$FACILITY, 'p=s' => \$PRIORITY, 'q+' => \$quiet);
#print "$SERVER_NAME\n$FACILITY\n$PRIORITY\n";
#Sys::Syslog::setlogsock('unix');
openlog ($SERVER_NAME,'ndelay',$FACILITY);
if (!($quiet)) {syslog($PRIORITY,"Logging Started -- Logger version 1.1");}
$| = 1;
while (<>) {
if (!($quiet)) {print $_ unless $_ =~ /^\s+$/};
chomp;
syslog($PRIORITY,$_) if $_;
}
closelog;
$| = 0;
Note : This is all regarding Linux and free software, as that's what I mostly use, but you should be fine with a syslog client on Windows to send the logs to a Linux syslog server.
Logging to an SQL server: With only ~30 machines, you should be fine with pretty much any centralised syslog-alike and an SQL backend. I use syslog-ng and MySQL on Linux for this very thing.
Pretty frontends for graphing are the main problem -- It seems that there is a lot of hacked-up front-ends which will grab items from the logs and show how many hits, alerts etc but I've not found anything integrated and clean. Admittedly this is the main thing that you're looking for... (If I find anything good then I'll update this section!)
Alerting: I use SEC on a Linux server to find bad things happening in the logs and alert me via various methods. It's incredibly flexible and not as clicky as Splunk. There's a nice tutorial here which guides through a lot of the possible features.
I also use Nagios for graphs of various stats and some alerting which I don't get from the logs (such as when services are down etc). This can be easily customized to add graphs of anything you like. I have added graphs of items such as the number of hits made to an http server, by having the agent use the check_logfiles plugin to count the number of hits in the logs (it saves the position it gets up to for each check period).
Overall, it depends on how much your time will cost to set this up, as there are many options which you can use but they aren't as integrated as Splunk and will probably require more effort to get doing what you want. The Nagios graphs are straightforward to set up but don't give you historical data from before you add the graph, whereas with Splunk (and presumably other front-ends) you can look back at the past logs and graph things you've only just thought of to look at from them.
Note also that the SQL database format and indexing will have a huge effect on the speed of queries, so your idea of fulltext indexing will make a tremendous increase to the speed of searches. I'm not sure if MySQL or PostgreSQL will do something similar.
Edit : MySQL will do fulltext indexing, but only on MyISAM tables prior to MySQL 5.6. In 5.6 Support was added for InnoDB.
Edit: Postgresql can do full text search of course: http://www.postgresql.org/docs/9.0/static/textsearch.html
Best Answer
Edit: 11/17/14
This answer may still work, but in 2014, using the Audisp plugin is the better answer.
If you are running the stock ksyslogd syslog server I don't know how to do this. But there are great instructions for doing it with rsyslog at their Wiki. ( http://wiki.rsyslog.com/index.php/Centralizing_the_audit_log )
I will summarize:
On the sending client (
rsyslog.conf
):Note that the
imfile
module will need to have been loaded previously in the rsyslog configuration. This is the line responsible for that:So check if it's in your
rsyslog.conf
file. If it's not there, add it under the### MODULES ###
section to enable this module; otherwise, the above configuration for auditd logging will not work.On the receiving server (
rsyslog.conf
):Restart the service (
service rsyslog restart
) on both hosts and you should begin receivingauditd
messages.