Solrcloud: Error 500 after restart

solrzookeeper

I have 3 vm windows, on all of them is installed solr 5.4.0 and 3.4.6 zookeeper; the two services start automatically when the vm start. After reboot vms the solr does not respond correctly, it gives me error500:

HTTP ERROR 500 Problem accessing /solr/. Reason: Server Error …
SolrException: Error processing the request. CoreContainer is either
not initialized or shutting down.

Checking the logs I have also seen this error:

o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: Error
occurred while loading solr.xml from zookeeper

Trying to stop the zookeeper and solr services on all vm and start again (before start zookeeper on all vm and after solr), sometimes takes solr starts to work properly; other times by solrcloud section of the UI, we notice that the nodes are in the down state (written brown)

I list the procedures that I run to create the cluster solr:

create folder "zookeeper/data" and changed the zoo.cfg file
by adding the zookeeper nodes in this way

initLimit=5

syncLimit=2

dataDir=C:/zookeeper/data

clientPort=2181

server.1=zknode1:2888:3888

server.2=zknode2:2888:3888

server.3=zknode2:2888:3888

created the myid files with the respective number on the nodes (1, 2,
3) in the zookeeper\data
by NSSMs I created zk service that starts the script
zookeeper\bin\zkServer.cmd, and the solr service that starts
solr\bin\solr.cmd with the following parameters

-c -z "zknode1:2181,zknode2:2181,zknode3:2181" -f

I create a collection based on an existing configuration on solr not cloud

solr.cmd create_collection -c mycoll -d C:\conf\mycoll\conf -n mycoll
-shards 1 -replicationFactor 3

on each solr node is see that is created mycoll_shard1_replicaX folder, where X changes on each node (1,2,3). Querying all works, even dataimport, but if I restart the nodes, finding the problem written above. I'm wrong in the creation of the collection, or is this a known bug?

Best Answer

Suggesting that maybe zookeeper isn't started before solr tries to initialise. Using nssm dependencies can you make solr service dependant on zookeeper service. https://nssm.cc/usage

Related Solutions

Solr 4.4 with tomcat 6 throws 503 on tomcat manager

The issue was probably with solr, since we had modified solrconfig initially and later even solr.xml (of solr) in the process of trial and error.

So, I did the whole thing again from ground zero. This time I referred another, rather relevant How-To

In sort, the mistake I made was to make changes to solr. Don't do that!
Solr Tomcat Wiki suggests: "Solr4.3 requires completely different deployment. These instructions are not current and are for an indeterminate version of Solr."
With wiki you tend to click on the index and miss fine print! I had referred it, therefore messed with working solr (lucky I had taken a back-up).

From the How-To I referred, I learned following things:

1 Don't make changes to a working solr config (i.e. run solr from jetty to check if it starts up fine)

2 Get tomcat up and running prior to adding solr in to it. (run both independently. Stop both the service if you are satisfied that both are fine)

3 Let's say solr path is /usr/local/solr-4.x, then copy solr-4.x.war from /usr/local/solr-4.x/dist/solr-4.x.x.war to /usr/local/solr-4.x/example/solr/solr.war . That's all you need to do in the solr directory. Rest, you need to take few files from solr directory in to tomcat directory (pt. 6)

4 Modify tomcat's server.xml and make sure you have URIEncoding in it:

<Connector port="8080" protocol="HTTP/1.1"
    connectionTimeout="20000"
    redirectPort="8443"
    URIEncoding="UTF-8" />

5 Create solr.xml in tomcat's conf : vim /usr/local/tomcat/conf/Catalina/localhost/solr.xml add following lines:

<Context path="/solr" docBase="/usr/local/solr/example/solr/solr.war" debug="0" crossContext="true">
        <Environment name="solr/home" type="java.lang.String" value="/usr/local/solr/example/solr" override="true"/>
</Context>

note: solr cores are under the path: /usr/local/solr/example/solr/core{0,1,..n}

6 Copy Solr extensions files and log4j configuration file to tomcat:

cp -v /usr/local/solr/example/lib/ext/*  /usr/local/tomcat/lib/
cp -v /usr/local/solr-oncars/example/resources/log4j.properties /usr/local/tomcat/conf/

For the rest, cross-check with the 'How-To'

In the end, grant tomcat ownership to solr and tomcat directories (if tomcat run as user/group 'tomcat'):

chown -R tomcat:tomcat /usr/local/tomcat
chown -R tomcat:tomcat /usr/local/solr

Start up tomcat and access the solr-url, it worked!

Related Topic