While running Logstash with a file
input against your logs on a CIFS share will work, I don't think it'll work very well. I haven't directly used Logstash like that but, In my experience using Logstash-Forwarder to watch logs files over an SSHFS mount, it doesn't deal well will file rotations or reboots of either end.
As for not being sure how to deal with your multi-line exceptions with Filebeat, I don't think you need to worry about it. FileBeat just takes
lines from the files you want to ship, and fires them across the network. It adds a few fields, but they don't affect the overall concept of FileBeat being a very basic log shipper.
This means you can just run your multi-line filter in Logstash on the collecting server, just as you would if you ran Logstash on the app servers directly.
Now, depending on your log volume, you might find that you need to increase the number of workers for LS to handle grokking your data effectively.
What I do to handle such things is very similar to your option 2, but instead of just having two LS instances ("Broker" and a "Parser"), I have 3.
+-------------------+
+-------------------+|
+-------------------+||
| App Servers |||
| +----------+ ||+
| | FileBeat | |+
+----+----------+---+
/
/
/
+----------------/----------------------------------------+
| / Collecting Server |
| +----------/-+ +---------------------+ +------------+ |
| | Logstash | | Logstash | | Logstash | |
| | Broker | |Multi-line Pre-Parser| | Parser | |
| +------------+ +---^-----------------+ +-----^---V--+ |
| | | | | | |
| | | Redis | | | |
| V +---------------------V------+ | | |
| +-------> DB0 | DB1 + --->+ | |
| +----------------------------+ / |
+-------------------------------------------------/-------+
/
/
/
+-------------------+ /
+-------------------+|/
+-------------------+||
| ElasticSearch ||+
| Cluster |+
+-------------------+
All the Pre-Parser
instance does is transform multi-line log entries into a single line so that the Parser
can do it's job properly. And even then, I'm checking type and tags to see if there's even a possibility that the line(s) will be multi-line, so the overhead is minimal.
I'm easily able to push 1000 events a second through it (barely hitting 20% CPU). Further, the system is an ELK stack-in-a-box, so with dedicated nodes for both LS and ES, it should be easy.
Why not just crank up the workers
on the Parser instance? Well, this stems from the fact that the multiline
filter in LS doesn't support multiple workers.
multiline
This filter will collapse multiline messages from a single source into one Logstash event.
The original goal of this filter was to allow joining of multi-line messages from files into a single event. For example - joining java exception and stacktrace messages into a single event.
Note: This filter will not work with multiple worker threads -w 2
on the Logstash command line.
Kibana uses "index patterns" to visualize the data stored in your elasticsearch indices.
You need to hit the elasticsearch restful endpoint and check what your indices are named by doing
curl -X GET <elasticsearchIP>:<elasticsearchport>/_cat/indices?v
This will list all the indices. Then, under kibana go to
management -> index patterns -> create index patterns
Here you write a regular expression that matches one or more of your elasticsearch indices. For example if your indices look like mine:
logstash-2018.10.29
logstash-2018.11.14
you could write an index pattern called
log*
and it would show data from from both of those logstash indices
Best Answer
If you followed the official Filebeat getting started guide and are routing data from Filebeat -> Logstash -> Elasticearch, then the data produced by Filebeat is supposed to be contained in a
filebeat-YYYY.MM.dd
index. It uses thefilebeat-*
index instead of thelogstash-*
index so that it can use its own index template and have exclusive control over the data in that index.So in Kibana you should configure a time based index pattern based on the
filebeat-*
index pattern instead oflogstash-*
. Alternatively you could run theimport_dashboards
script provided with Filebeat and it will install an index pattern into Kibana for you. The path to theimport_dashboards
script may vary based on how you installed Filebeat. This is for Linux when installed via RPM or deb./usr/share/filebeat/scripts/import_dashboards -es http://localhost:9200
You can check if data is contained in a
filebeat-YYYY.MM.dd
index in Elasticsearch using a curl command that will print the event count.curl http://localhost:9200/filebeat-*/_count?pretty
And you can check the Filebeat logs for errors if you have no events in Elasticsearch. The logs are located at
/var/log/filebeat/filebeat
by default on Linux. You can increase verbosity by settinglogging.level: debug
in your config file.