Linux – Pipe directly to Logstash

Your post doesn't describe much in the way of specs (memory on the LS indexer, log volume or much else) but I'll try and answer your questions best I can first. Disclaimer: I'm one of the logstash devs -

1. Apache going nuts was likely a side effect of the logstash process acting up. I'd put that aside for now.

2. The sane way to make ES f/b/s is add more ES nodes. It's seriously that easy. They even autodiscover each other depending on network topology. After 17 years in this industry I've never seen anything scale horizontally as easy as ElasticSearch.

3. To f/b/s Redis, don't use any redis clustering. Newer versions of Logstash can do Redis loadbalancing internally. The Redis output supports a list of Redis hosts in the plugin config and support is about to be added to the input side as well to match that. In the interim you can use multiple Redis input definitions on the indexer/consumer side.

4. I can't answer this beyond saying that it sounds like you're trying to do to much with a single (possibly underpowered host).

Any good scaling process starts with breaking collocated components into distinct systems. I don't see your configs gist'd anywhere but the places where logstash 'bottlenecks' are in filters. Depending on how many transformations you're doing it can have an impact on the memory usage of Logstash processes.

Logstash works a lot like legos. You can either use a 2x4 brick or you can use two 2x2 bricks to accomplish the same task. Except in the case of logstash, it's actually sturdier to use smaller bricks than a single big brick.

Some general advice we normally give is:

• ship logs as quickly as possible from the edge If you can use pure network transport instead of writing to disk, that's nice but not required. Logstash is JVM-based and that has good and bad implications. Use an alternate shipper. I wrote a python based one ( https://github.com/lusis/logstash-shipper ) but I would suggest that folks use Beaver instead ( https://github.com/josegonzalez/beaver ).

• generate your logs in a format that requires as little filtering as possible (json or optimally json-event format) This isn't always possible. I wrote a log4j appender to do this ( https://github.com/lusis/zmq-appender ) and eventually broke out the pattern layout into its own repo ( https://github.com/lusis/log4j-jsonevent-layout ). This means I don't have to do ANY filtering in logstash for those logs. I just set the type on input to 'json-event'

For apache, you can try this approach: http://cookbook.logstash.net/recipes/apache-json-logs/

• break things into multiple components In every talk I've done about logstash, I describe it as a unix pipe on steroids. You can make the pipeline as long or as short as you like. You scale logstash by shifting around responsibilities horizontally. This might mean making the pipeline longer but we're not talking about anything statistically relevant in terms of latency overhead. If you have greater control over your network (i.e. NOT on EC2) you can do some amazing things with standard traffic isolation.

Also note that the logstash mailing list is VERY active so you should always start there. Sanitize and gist your configs as that's the best place to start.

There are companies (like Sonian) scaling ElasticSearch to petabyte levels and companies (like Mailchimp and Dreamhost) scaling Logstash to insane levels as well. It can be done.

The 'Tools that works with Graphite' page http://graphite.readthedocs.org/en/latest/tools.html list several. I have tried the PowerShell script 'Graphite PowerShell Functions' https://github.com/MattHodge/Graphite-PowerShell-Functions and it works well.

edit I mis-read your question, you were talking only about Logstash and Kibana but not about Graphite. I don't use Logstash+Kibana for system metrics, but I use Statsd+Graphite. So not sure if my answer is valid to you but if you use the Graphite Logstash input http://logstash.net/docs/1.4.2/ you could use these tools.