I have a few servers running on AWS and have Nagios/Icinga doing the monitoring of all critical services.
We're trying to figure out the best way to monitor all logs – system, DB, PHP, Apache, etc
– on the system so we know about issues (for e.g. that Apache reached the max_clients
threshold yesterday) immediately via email. We only look at logs currently after a service goes down, not before, which is bad.
I'm new to Linux administration and I've identified the following options after a search online:
- Nagios scripts to monitor logs – The problem is most of them check one log file for one specific regex at a time. It's not scalable to install one service for each log file (I don't even know all the log files we have to monitor!)
- A service such as logrobot.com – I'm not sure how effective this is though.
Appreciate your advice on what's the best way to monitor all these logs on multiple servers with minimal configuration.
Best Answer
Don't know how much servers/logs you have to monitor but there are many solutions out there
small environment
Use rsyslog and a frontend you like (ex. LogAnalyzer http://loganalyzer.adiscon.com/)
bigger environment
We monitor our serverlogs from (+300 system) with beaver as logshipper, logstash as indexer and elasticsearch as backend. This solution scales up to [insert random number here] hosts ;)