Java – Solr Out of memory exception

javalucenesolrtomcat

Solr is giving out of memory exception. Index size is small roughly around 30GB. Full Indexing was completed fine. Later while searching maybe when it tries to load the results in memory it starts giving this exception. Though with the same memory allocated to Tomcat and exactly same solr configuration replica on another server it is working perfectly fine. I am working on 64 bit software's including Java & Tomcat on Windows. With smaller index size it is working perfectly fine. I was thinking of increasing the tomcat heap space allocated but then how come on a different server with exactly same system and solr configuration it is working fine?
Any help would be appreciated.

Here are the logs:

The server encountered an internal error (Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: false in null ————————————————————- java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068) at org.apache.solr.core.SolrCore.(SolrCore.java:579) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4072) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4726) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.index.SegmentTermEnum.termInfo(SegmentTermEnum.java:180) at org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:91) at org.apache.lucene.index.SegmentReader$CoreReaders.(SegmentReader.java:122) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:613) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:104) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:27) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:74) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69) at org.apache.lucene.index.IndexReader.open(IndexReader.java:476) at org.apache.lucene.index.IndexReader.open(IndexReader.java:403) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057) at org.apache.solr.core.SolrCore.(SolrCore.java:579) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4072) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4726) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) ) that prevented it from fulfilling this request.

Best Answer

I think you should be treating this the same as any other OOME problem:

It could be that your application just needs some more memory - Fix: increase the max heap size using the JVM -Xmx option.
It could be that your application has a memory leak. (Possibly in Solr, or possibly in your code.) - Fix: use a memory profiler to identify the leak and fix it.

... but then how come on a different server with exactly same system and solr configuration it is working fine?

Clearly they are not the same. Something is different, but you can't figure out what it is. (But I don't think that finding out what the difference is a good way to solve the problem.)

Related Solutions

Java – Solr – with LocalSolr for GIS Support Installation problems

java.lang.NoClassDefFoundError: org/apache/lucene/spatial/tier/DistanceFilter

A NoClassDefFoundError basically means that the mentioned class is missing in the runtime classpath, while it was available in the classpath during compiletime of the API in question (that is the difference with ClassNotFoundException).

To solve this problem you obviously need to place the mentioned class (or the JAR file which includes the mentioned class) in the runtime classpath.

As the package name already hints, this concerns Apache Lucene which is available at http://lucene.apache.org. Do you have its libraries in the runtime classpath? If so, are you sure that there are no collisions between duplicated classes/jars of different versions spreading over the classpath (i.e. in WEB-INF/lib, Appserver/lib and JRE/lib) which might be loaded by different classloaders? That can namely also cause this kind of errors.

Java – How to create a memory leak in Java

Here's a good way to create a true memory leak (objects inaccessible by running code but still stored in memory) in pure Java:

The application creates a long-running thread (or use a thread pool to leak even faster).
The thread loads a class via an (optionally custom) ClassLoader.
The class allocates a large chunk of memory (e.g. new byte[1000000]), stores a strong reference to it in a static field, and then stores a reference to itself in a ThreadLocal. Allocating the extra memory is optional (leaking the class instance is enough), but it will make the leak work that much faster.
The application clears all references to the custom class or the ClassLoader it was loaded from.
Repeat.

Due to the way ThreadLocal is implemented in Oracle's JDK, this creates a memory leak:

Each Thread has a private field threadLocals, which actually stores the thread-local values.
Each key in this map is a weak reference to a ThreadLocal object, so after that ThreadLocal object is garbage-collected, its entry is removed from the map.
But each value is a strong reference, so when a value (directly or indirectly) points to the ThreadLocal object that is its key, that object will neither be garbage-collected nor removed from the map as long as the thread lives.

In this example, the chain of strong references looks like this:

Thread object → threadLocals map → instance of example class → example class → static ThreadLocal field → ThreadLocal object.

(The ClassLoader doesn't really play a role in creating the leak, it just makes the leak worse because of this additional reference chain: example class → ClassLoader → all the classes it has loaded. It was even worse in many JVM implementations, especially prior to Java 7, because classes and ClassLoaders were allocated straight into permgen and were never garbage-collected at all.)

A variation on this pattern is why application containers (like Tomcat) can leak memory like a sieve if you frequently redeploy applications which happen to use ThreadLocals that in some way point back to themselves. This can happen for a number of subtle reasons and is often hard to debug and/or fix.

Update: Since lots of people keep asking for it, here's some example code that shows this behavior in action.

Best Answer

Related Solutions

Java – Solr – with LocalSolr for GIS Support Installation problems

Java – How to create a memory leak in Java

Related Topic