At what point do I want to "upgrade" from the native Magento search engine to something like Apache SOLR, or a commercial product? Are there specific features I'd be looking for, or is it simply a function of traffic, SKU count, or something else?
Magento Search vs. SOLR – Which is Better?
searchsolr
Related Solutions
I work a lot with Solr and created a Extension for that here http://solrgento.com. You can search over attributes and CMS pages. I had the same idea as Ivan and implemented the whole catalog view over Solr. Its really freaking fast ;-)
As you said, Solr and Elasticsearch are both built on top of the Apache Lucene project, so what's the difference and why would Magento deprecate support for Solr in favor of Elasticsearch?
First, if you're not familiar with these tools, they are open-source search engines. They support text search (think keywords typed into your search box) among other things. Honestly, these are serious search engines and if you are planning to manage either Solr or Elasticsearch yourself, you may want to try it first as these tools are significant pieces of infrastructure. That said, they are seriously powerful and can support more nuanced search scenarios than your default full-text search in a relational database like Postgres.
The feature set of Solr and Elasticsearch is, for practical purposes, identical. Solr really excels at some more advanced search problems and has a pretty natural extension framework. Elasticsearch favors a JSON/REST API approach rather than Solr's DSL-oriented interface (although Solr is releasing more JSON/REST APIs).
For differences, Elasticsearch is maybe a little easier to use out-of-the-box, but it's not a consumer product. In fact, the simple starting cases with Elasticsearch can lead to later difficulties for newbies who may not appreciate what they're getting into. For single server instances, I would say they're roughly equivalent.
As to why Magento is pushing Elasticsearch over Solr, I think Elasticsearch's JSON interface is a little easier to learn quickly, and its deployment story requires less operational skill than Solr.
In either case, if you're considering a different search engine for Magento you should consider some of the managed services for Elasticsearch (or Solr) rather than managing the servers yourself, as that is a whole skill in itself.
Best Answer
The distinguishing features of using Solr over the default Magento MySQL Fulltext engine are:
The community seems to lean toward Solr in general, though other search connectors have been made available.
3rd party search:
3rd party search products work in one of two ways: they replace your search and category pages with their own versions and typically host them under a subdomain on your site so as not to interfere with SEO (e.g. SLI), or, they iFrame search into your site (Picosearch). In my opinion I don't want my stores at the mercy of a 3rd party hosting on AWS; or worse yet on their own in-house datacenter.
I recently met a vendor wearing a funny hat who purportedly had a custom search tool who plugs into the Magento Solr connector but it runs as a SaaS cloud service with a control panel. This is not normally how the 3rd party search integrations work - I'm curious what the performance is like.
Timing/Deployment
There is no formula as to determine when to make the switch - when you should migrate is entirely up to you. Keep in mind it isn't a two-second implementation - your theme/plugins (especially any navigation plugins) may require tweaking or upgrading.
It's not going to give you miracle results overnight - like everything else in running your business it'll only be as good as the amount of time you invest into it. You will still likely have to have indexed fields with keywords you want to optimize for. You'll have to play with the weighting settings heavily, too. It's not an overnight fix.
A note about performance
The distinguishing feature of the Solr integration over fulltext really is that Magento can utilize Solr to offload the layered navigation and category page results rendering to it. Because Solr is so highly optimized it incurs very little overhead to do so. This is handy to mitigate traffic spikes and to spread your load across more than just your Apache servers.