Magento 1.7 – How to Get SOLR Working on Magento CE 1.7.0.2

How to get SOLR working on magento ce 1.7.0.2

Why i need SOLR:
I need to use SOLR to reduce my +10seconds loading time on searches. I know there is also sphinx, but i don't like that one.

I have followed instructions from here:

install SOLR:
http://inchoo.net/ecommerce/install-apache-solr/

what to do next:
magentocommerce.com/boards/viewthread/301508/ (i followed the 2nd post, but i cannot find the conf from step5. i assume i only need to use the schema.xml from the extension and paste it into the conf folder in the SOLR installation)

used this extension:
code.google.com/p/magento-community-edition-solr/

Magento version: 1.7.0.2
SOLR version: 4.6.0
Running on VPS (CentOS 6.4)
magento-community-edition-solr: 1.1.4

SOLR is running and i can access the Solr Dashboard. I have set the settings in magento system/configuration/Solr Search Engine to my settings like host/path and port.

Problem:
I re-index the catalog search indexes and here starting the troubles. I can see activity on my server (mysql. i assume data is re-indexed into the magento database) but when indexes are re-indexed in the sql database i assume it needs to re-index data to/for Solr, right? The "in progress" won't stop and i see nothing been added to Solr collection.
When i use the searchbox on my site i get a 400 error "400' Status: Bad Request" I also see some activity in putty when i search on my site, so i think the connection between magento and Solr are OK. If there is no connection i assume Solr cannot do stuff whn i trigger the magento search. No results found because the 400 error so i get the default magento search results.

What i think:
I think the problem is in the conf folder (schema.xml). When i use the default schema.xml i get the error "undefined filed store_id". So i need to map the store_id, i know. So i use the schema.xml that i can find in the solr magento extension. When i upload that file i only get more errors and Solr won't work at all.

What i have done:
Replaced and edited schema.xml over and over again. Searched google and found people with the same problem but no answers for this problem.

More questions:
When i installed Solr i find out that there is a collection1 "core selector". I don't see anything about that in the install instruction of Solr or the Solr extension. Maybe because my Solr version is newer as used in the tutorials? BUt that collection1 i use to store the conf folder with the schema.xml file. In the tutorials people talk to paste conf folder into example folder, but my structure is example/collection1/conf. Am i wrong?
How does magento knows what collection it need to use? I think here is something i need to edit (or not) or do i need to add the collection1 to the Solr path i use in the magento extension configuration?

Thanks

edit 1:
I have installed (uploaded) solr version 3.4.0 and transfered the schema.xml from the magento extension to the conf folder from solr 3.4.0. I already see 1 problem. The standard schema.xml shows this <schema name="example" version="1.4"> and the schema from magento extension <schema name="example" version="1.5"> i ignored that and headed on. Solr errors and won't start. I removed the new schema and use the standard schema. Error when i start solr "store_id is not a fallid field or something". So i have put the filed code from magento schema.xml into the solr schema.xml.
Yes! Search is FAST! But shows no results. hits=0 status=0
I need to tell that the reindex from the catalog search is still running/processing and won't finish and i don't see documents or data added to the solr data/index folder (if that is where documnets are stored).

Another problem is that solr now only runs when i keep the shell connection open. When i close putty, solr will stop. I think this can be solver by using a other command to start solr. Maybe this is a red flag for a other problem, so i mentioned this 😉

edit 2:
I have installed solr for the 3th time, now with this instructions:
ctdev.co.za/magento-solr-search-setup-for-community-edition-solr-ce/#comment-86

WHen i reindex the search catalog by shell i get this after a couple minutes:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 7 bytes) in /home/admin/domains/test.com/public_html/lib/Zend/Db/Statement/Pdo.php on line 253

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 7 bytes) in /home/admin/domains/test.com/public_html/lib/Zend/Db/Statement/Pdo.php on line 253

Is this error related to the solr index problem?
It still won't put data into solr 🙁

I have enlarged the memory and now i see this when re-indexing:

Jan 26, 2014 5:59:30 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
        commit{dir=/opt/solr/solr/data/index,segFN=segments_3,version=1390755510203,generation=3,filenames=[segments_3]
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510203
Jan 26, 2014 5:59:30 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {deleteByQuery=*:*} 0 6
Jan 26, 2014 5:59:30 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=6
Jan 26, 2014 5:59:30 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true,expungeDeletes=false)
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
        commit{dir=/opt/solr/solr/data/index,segFN=segments_3,version=1390755510203,generation=3,filenames=[segments_3]
        commit{dir=/opt/solr/solr/data/index,segFN=segments_4,version=1390755510204,generation=4,filenames=[segments_4]
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510204
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        filterCache{lookups=2,hits=1,hitratio=0.50,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        queryResultCache{lookups=2,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3c76e596 main from Searcher@39eb305e main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3c76e596 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@3c76e596 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@39eb305e main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=2,hits=1,hitratio=0.50,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
        queryResultCache{lookups=2,hits=0,hitratio=0.00,inserts=2,evictions=0,size=2,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {commit=} 0 230
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=230
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true,expungeDeletes=false)
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
        commit{dir=/opt/solr/solr/data/index,segFN=segments_4,version=1390755510204,generation=4,filenames=[segments_4]
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1390755510204
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1c6b7657 main from Searcher@3c76e596 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1c6b7657 main
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher@1c6b7657 main
Jan 26, 2014 5:59:31 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@3c76e596 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=1,cumulative_hitratio=0.50,cumulative_inserts=1,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=2,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jan 26, 2014 5:59:31 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {optimize=} 0 12
Jan 26, 2014 5:59:31 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=12

And now i get this error (i'm almost there ;-)):

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 72243474 bytes) in /home/admin/domains/test.com/public_html/lib/Apache/Solr/HttpTransport/FileGetContents.php on line 146

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 72243474 bytes) in /home/admin/domains/test.com/public_html/lib/Apache/Solr/HttpTransport/FileGetContents.php on line 146

Memory in php.ini is now 2GB. i think that is to much but i get better results. Here they come:

INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=13
Jan 26, 2014 6:47:01 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[17637854]} 0 29
Jan 26, 2014 6:47:01 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=17637854] unknown field 'product_id'
        at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:340)
        at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
        at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:157)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Jan 26, 2014 6:47:01 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=400 QTime=29
Catalog Search Index index was rebuilt successfully
[root@server01 shell]# Jan 26, 2014 6:49:51 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select params={fl=product_id,score&start=0&q=jeans&json.nl=map&wt=json&fq=store_id:1&rows=100} hits=0 status=0 QTime=2

I cannot find the data in the solr data index folder. Meybe i need to look somewhere else, but when i search on the shop there are no results 🙁