.net – How to get more out of Lucene.net

I'm not sure if this is the correct way but you could add another non nullable term to the index (say BookIsPublished) with a default value and if no search term is entered by the user return all matching items on the BookIsPublished constant field.

Lucene

Lucene and the .NET port, Lucene.Net, is a search engine library for supporting full-text search in an application; it builds an inverted index based on the Document (and the fields within the Document) that you feed it to support full-text search. An example of this is search within the Nuget Gallery source, where a nuget package and its properties is converted to a document to pass to Lucene. The inverted index is stored across files within a directory.

Elasticsearch

Elasticsearch is a distributed search engine that uses Lucene under the covers - An Elasticsearch cluster can be made up of one or more nodes, where each node can contain a number of shards and replicas; each shard is a complete Lucene index. Having such infrastructure enables fast performance and allows horizontal scaling to handle search across a large amount of data since you are no longer limited by the constraints of a single Lucene index on a single machine. In addition you can achieve high availability with fault tolerance and disaster recovery since data can be replicated across shards meaning there is no single point of failure. An example of Elasticsearch with NEST is up on my blog.

Which to use?

Well, it depends on your use case (it nearly always does, right?); if your application is one that gets installed onto a machine and all data is persisted locally, you might decide to use Lucene library within the application and persist the index directory to local disk. Similarly, if you have a simple web application that runs on a single server with a small number of users then using Lucene may also be a sensible choice. On the other hand, if your application runs across multiple machines in a web farm and requires search capabilities, going with a distributed search engine like Elasticsearch would be a good idea.

How well does Elasticsearch scale? Back in 2013, Github was using Elasticsearch to index 2 billion documents i.e. all the code files in every repository on the site - across 44 separate Amazon EC2 instances, each with two terabytes of ephemeral SSD storage, giving a total of 30 terabytes of primary data. Stackoverflow also uses Elasticsearch to power search on this site (perhaps a dev could comment with some figures/metrics?)