I'm trying to setup a central logstash configuration. However I would like to be sending my logs through syslog-ng and not third party shippers. This means that my logstash server is accepting via syslog-ng all the logs from the agents.
I then need to install a logstash process that will be reading from /var/log/syslog-clients/* and grabbing all the log files that are sent to the central log server. These logs will then be sent to redis on the same VM.
In theory I need to also configure a second logstash process that will read from redis and start indexing the logs and send them to elasticsearch.
My question:
Do I have to use two different logstash processes (shipper & server) even if I am in the same box (I want one log server instance)? Is there any way to just have one logstash configuration and have the process read from syslog-ng —> write to redis and also read from redis —> output to elastic search ?
Diagram of my setup:
[client]——-syslog-ng—> [log server] —syslog-ng <—-logstash-shipper —> redis <—-logstash-server —-> elastic-search <— kibana
Best Answer
I'm not sure if I understood the question correctly, but I do know that Syslog-NG can ship directly to Logstash without the need for an additional shipper as an intermediary. You could define a destination in syslog-ng.conf similar to this example:
And then define a log action to send Syslog messages from
source s_src
todestination
:Which should enable the message transmission. Don't forget to restart the syslog-ng service to apply the changes.
source: The Logstash Book