elasticsearch – How Well Does Elasticsearch Compress Data?

compressionelasticsearch

I am looking to scope the servers required for an elasticsearch proof-of-concept.

Ultimately, my question is this:

Given 1GB of json text indexed by elasticsearch, how much can I expect to be occupied on disk by elasticsearch?

Obviously there are many variables, but I'm going for orders of magnitude. 100MB? 100GB?

I understand that elasticsearch performs compression ( http://www.elasticsearch.org/guide/reference/index-modules/store/ ), but I don't know what kind of footprint the indexes and other structures occupy.

Anecdotal answers are acceptable, but please also let me know what version you're using.

Best Answer

The answer is: it depends.

A blog post by Adrien Grand, who works on Elasticsearch did some benchmarking with this from the Lucene standpoint. It looks like he was getting about 2x improvement.

He also mentions LUCENE-4226, where some of this underlying compression work was done. There's a few benchmarks listed within, comparing the various compression algorithms tested.

As well, based on this Elasticsearch 0.19.5 release announcement, it appears that store-level compression defaults to LZF, with Snappy coming some time in the future. Further looking around showed that Snappy experimental support appeared in 0.19.9

Related Solutions

Linux – How to Zip/Compress a Symlink

You can store symlinks as symlinks (as opposed to a copy of the file/directory they point to) using the --symlinks parameter of the standard zip.

Assuming foo is a directory containing symlinks:

zip --symlinks -r foo.zip foo/

Rar equivalent:

rar a -ol foo.rar foo/

tar stores them as is by default.

tar czpvf foo.tgz foo/

Note that the symlink occupies almost no disk space by itself (just an inode). It's just a kind of pointer in the filesystem, as you probably know.

Elasticsearch standalone mode with logstash reject data after five days

I've seen this before, and it's all to do with the amount of Heap Space allocated to Logstash by the JVM.

You can try increasing this by providing this flag -XX:MaxHeapSize=256m to the JVM when you start it, except you should probably try setting the MaxHeapSize to something like 512m or even bigger.

Elasticsearch comes with some pretty sane defaults, but it's possible to do even more tuning, setting the size of it's heap and so on for searching.

ES and Logstash can scale to billions of events, and many terabytes of log data, with careful configuration.

Best Answer

Related Solutions

Linux – How to Zip/Compress a Symlink

Elasticsearch standalone mode with logstash reject data after five days

Related Topic