Multiple instances of Logstash + Elasticsearch on AWS

Usual suspects for ES with Kibana are :

• *Too small amount of memory available for ES** (which you can investigate with any probe system such as Marvel, or something that will send you JVM data outside the VM for monitoring)
• Long GC durations (turn on GC logging and see if they do not happen when the ES stop responding)

Also the "usual" setup for ES is 3 servers to allow better redundancy when one server is down. But YMMV.

You can try the new G1 garbage collector too, which has (in my case) a much better behavior than CMS in my Kibana ES.

The GC duration problem is usually the one that happens when you're looking somewhere else and will typically lead to a loss of data because ES stops responding.

Good luck with these :)

After a few more frustrating days I have concluded that the json/json_lines codec is broken - possibly only when used with tcp inputs.

However, I found a workaround, using a filter:

filter {
  if ("sensu" in [tags]) {
    json {
      "source" => "message"
    }
  }
}

This, and a few mutations produces the effect I was originally trying to achieve. For posterity, here's my working logstash.conf which combines logs and cpu/memory metrics data from sensu:

input {
  file {
    path => [
      "/var/log/messages"
      , "/var/log/secure"
    ]
    type => "syslog"
    start_position => "end"
  }

  file {
    path => "/var/log/iptables"
    type => "iptables"
    start_position => "end"
  }

  file {
    path => ["/var/log/httpd/access_log"
        ,"/var/log/httpd/ssl_access_log"
    ]
    type => "apache_access"
    start_position => "end"
  }

  file {
    path => [
      "/var/log/httpd/error_log"
      , "/var/log/httpd/ssl_error_log"
    ]
    type => "apache_error"
    start_position => "end"
  }

  lumberjack {
    port => 5043
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }

  tcp {
    host => "localhost"
    port => 9250
    mode => "server"
    tags => ["sensu"]
  }

}

filter {
  if ("sensu" in [tags]) {
    json {
      "source" => "message"
    }
    mutate {
      rename => { "[check][name]" => "type" }
      replace => { "host" => "%{[client][address]}" }
      split => { "[check][output]" => " " }
      add_field => { "output" => "%{[check][output][1]}" }
      remove_field => [ "[client]", "[check]", "occurrences" ]
    }
  } else if([type] == "apache_access") {
    grok {
      match => { "message" => "%{IP:client}" }
    }
  }
}

filter {
  mutate {
    convert => { "output" => "float" }
  }
}

output {
  elasticsearch {
    host => "localhost"
    cluster => "webCluser"
  }
}

Unrelated to issue: The "output" is received as multiple values separated by spaces, hence the "split" operation. The second element is used and then converted to float, so that Kibana graphs it nicely (something I learned the hard way).