I've built a system that needs to collect new apache logs from four machines every hour, merge them, process them and store them. For the task of collecting the new logs I've written a bunch of cron script that reads in the line of already sent lines and start sending only the new lines, then store the new line number. This is synchronized with a merging script using some .lock files.
Although this is working quite well I would like to substitute this system with a more reliable system with good error handling and perhaps autodetect of new log files (they are rotated every day). Is there something I could use to do this kind of task?
Thank you!
Best Answer
There are many solutions for collecting/monitoring logs.
Some of them: syslog, syslog-ng, logsurfer, splunk... and I think much more. But I will recommend syslog-ng.
It is very configurable (for example he can store logs in database wich sometimes is very useful), elastic and do all you want.
With syslog-ng and MongoDB you can make quite fun stuff using mojology(A simple application to browse syslog messages stored in MongoDB, as logged by syslog-ng).
You can use any of this software.
Zabbix can collect logs but it's much more harder to configure what you want.
Syslog is not bad idea but sysylog-ng has much more features. Here is comparison of syslog and syslog-ng http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/features/comparison
So I would choose syslog-ng :)