I have a web application that is split over a couple of sites in the US and the UK. When we have issues I would like to be able to view the collated error logs from the 2 sites.
So I was thinking about doing this
1)setting up a splunk server at each site
2)setting 1 as a forwarder to the other
3)forwarding all error syslogs and warnings from site 1 to site 2
and when there was a problem, I would import all the apache logs from site 1 into the local splunk at it would forward them to site 2 for analysis with all the existing logs.
Q1) Is this reasonable using the splunk free license?
Q2) is it reasonable to send server syslogs that are at the error and warning level over the internet? ie performance and bandwidth wise
Best Answer
The best is to add a third machine (not very powerful) dedicated to log collection and collect the logs there. It's simple / secure / efficient etc. Otherwise you get overhead in both machines (sending , receiving).
What's the average log traffic you have ? is use splunk free and it's very good. the limit is that you can index only 500 MB/day and you don't have all the feature enabled. But just for debugging it's perfect.
Splunk free unfortunately doesn't support distributed search. But if you do what i told you before, you can index the logs of both sites in one splunk server, and having cross correlation.
Also, if you use rsyslog as the syslog manager, you have too many features that i doubt if splunk support them (Let's be honest ... splunk it's more dedicated to log analysis rather than log transfer )