How best to monitor logstash

elasticsearchlogstashmonitoring

I've seen this question on the mailing list a few times but haven't had a satisfactory answer.

How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch.

Logstash and especially elasticsearch are prone to resource starvation. They are both fantastic at picking up where they left off but how, exactly, are people watching their watchers?

Opinions welcome.

Best Answer

Personally i actually check that redis is still dequeuing on the central logging host, which is upstream of LS+ES.

i.e: redis-cli llen logstash is less than some fixed number.

This may not indicate that logs are appearing in redis at all though, but that could be checked too i guess.

Something like checking that redis-cli info | grep total_commands_processed keeps increasing, maybe ?

Related Solutions

Using Logstash as shipper

SOLVED:

I solved it by adding codec => "json" to my lumberjack output and input.

Output:

output {

    lumberjack {
            hosts => ["xx.xx.xx.xx"]
            port => 4545
            ssl_certificate => "./logstash.pub"
            codec => "json"
}

Input:

input { 
    lumberjack {
        port => 4545
        ssl_certificate => "/etc/ssl/logstash.pub"
        ssl_key => "/etc/ssl/logstash.key"  
        codec => "json"
  }
}

Linux – Logstash: Failed to flush outgoing items

To re-import date, remove $HOME/.sincedb file, remove indexes in elastic search , and set start_position to beginning:

 input {
 #apache
 file {
     type => "apache-access"
     path => "/var/log/apache2/access.log"
     start_position => beginning
 }

Best Answer

Related Solutions

Using Logstash as shipper

Linux – Logstash: Failed to flush outgoing items

Related Topic