Java 8 Metaspace: OutOfMemoryError: Dealing with reflection Inflation

garbage-collectionjavaout-of-memoryreflectionspring

There are lots of links and open Q&A s around web, still I'm missing a lot of information.

First things first

Issue:

java.lang.OutOfMemoryError: Metaspace

jvm:

java version:  "1.8.0_131"
vm:             Java HotSpot(TM) 64-Bit Server
vm args:       -Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=128m

Frameworks:

spring, hibernate, wicket, jetty

Suspect 1:

Over the period of usage, the metaspace grows gradually and the following reflection classes are proportionally loaded into metaspace [observed by jmap -histo cron jobs]

sun.reflect.GeneratedConstructorAccessor1299
sun.reflect.GeneratedMethodAccessor6929
sun.reflect.GeneratedSerializationConstructorAccessor4220

Possible Solution:

a. As we are using libraries that are heavily dealing with reflection stuffs, we think 128m is quite not enough to hold all the generatedXX classes in metaspace. So we are planning to double the metaspace limit. -XX:MaxMetaspaceSize=256m

b. We are not thinking of setting the following

-D sun.reflect.noInflation

-D sun.reflect.inflationThreshold

Suspect 2:

The Full GC is running continuously even before it reaches/occupies the full configured metaspace(128m) and the application becomes unresponsive/slow/sometime OOM as the jvm only does the FGC.

[Full GC (Metadata GC Threshold) [PSYoungGen: 224K->0K(698368K)]
[ParOldGen: 52910K->52933K(1398272K)] 53134K->52933K(2096640K),
[Metaspace: 92733K->92733K(1163264K)], 0.1964143 secs] [Times:
user=0.59 sys=0.00, real=0.19 secs]

Metaspace used 147414K, capacity 155731K, committed 159616K,
reserved 1187840K
class space used 17242K, capacity 19252K,
committed 20352K, reserved 1048576K

Possible Solution:

-XX:CompressedClassSpaceSize is not explicitly mentioned at the vm startup which may possibly cause over reserving the address space which leads to misleading committed sapce and hence the Full GC. So explicitly setting
-XX:CompressedClassSpaceSize=256m will help vm in correct memory planning & reserving.

Questions:

Suspect 1: Did someone face similar issue and got any fix?
Suspect 2: Is setting -XX:CompressedClassSpaceSize really makes a difference in metaspace planning/reserving and affects GC? Any pointers?
Any other suspects? recommendations?

Best Answer

After spending so much of time, it turns out there is no class leak and we are the victim of

"Reflection Inflation"

package sun.reflect;

/**
 The master factory for all reflective objects, both those in
 java.lang.reflect (Fields, Methods, Constructors) as well as their
 delegates (FieldAccessors, MethodAccessors, ConstructorAccessors).
**/

public class ReflectionFactory {


    // "Inflation" mechanism. Loading bytecodes to implement Method.invoke() and Constructor.newInstance() currently costs
    // 3-4x more than an invocation via native code for the first invocation (though subsequent invocations have been benchmarked
    // to be over 20x faster). Unfortunately this cost increases startup time for certain applications that use reflection
    // intensively (but only once per class) to bootstrap themselves. To avoid this penalty we reuse the existing JVM entry points
    // for the first few invocations of Methods and Constructors and then switch to the bytecode-based implementations.

I verified it personally, at every 16th time of refleciton generation this technique was used to improve the response time.

Finally we increase the metaspace and everything seems to work fine.

Related Solutions

Dealing with “java.lang.OutOfMemoryError: PermGen space” error

The solution was to add these flags to JVM command line when Tomcat is started:

-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

You can do that by shutting down the tomcat service, then going into the Tomcat/bin directory and running tomcat6w.exe. Under the "Java" tab, add the arguments to the "Java Options" box. Click "OK" and then restart the service.

If you get an error the specified service does not exist as an installed service you should run:

tomcat6w //ES//servicename

where servicename is the name of the server as viewed in services.msc

Source: orx's comment on Eric's Agile Answers.

Java – Change private static final field using Java reflection

Assuming no SecurityManager is preventing you from doing this, you can use setAccessible to get around private and resetting the modifier to get rid of final, and actually modify a private static final field.

Here's an example:

import java.lang.reflect.*;

public class EverythingIsTrue {
   static void setFinalStatic(Field field, Object newValue) throws Exception {
      field.setAccessible(true);

      Field modifiersField = Field.class.getDeclaredField("modifiers");
      modifiersField.setAccessible(true);
      modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);

      field.set(null, newValue);
   }
   public static void main(String args[]) throws Exception {      
      setFinalStatic(Boolean.class.getField("FALSE"), true);

      System.out.format("Everything is %s", false); // "Everything is true"
   }
}

Assuming no SecurityException is thrown, the above code prints "Everything is true".

What's actually done here is as follows:

The primitive boolean values true and false in main are autoboxed to reference type Boolean "constants" Boolean.TRUE and Boolean.FALSE
Reflection is used to change the public static final Boolean.FALSE to refer to the Boolean referred to by Boolean.TRUE
As a result, subsequently whenever a false is autoboxed to Boolean.FALSE, it refers to the same Boolean as the one refered to by Boolean.TRUE
Everything that was "false" now is "true"

Caveats

Extreme care should be taken whenever you do something like this. It may not work because a SecurityManager may be present, but even if it doesn't, depending on usage pattern, it may or may not work.

JLS 17.5.3 Subsequent Modification of Final Fields

In some cases, such as deserialization, the system will need to change the final fields of an object after construction. final fields can be changed via reflection and other implementation dependent means. The only pattern in which this has reasonable semantics is one in which an object is constructed and then the final fields of the object are updated. The object should not be made visible to other threads, nor should the final fields be read, until all updates to the final fields of the object are complete. Freezes of a final field occur both at the end of the constructor in which the final field is set, and immediately after each modification of a final field via reflection or other special mechanism.

Even then, there are a number of complications. If a final field is initialized to a compile-time constant in the field declaration, changes to the final field may not be observed, since uses of that final field are replaced at compile time with the compile-time constant.

Another problem is that the specification allows aggressive optimization of final fields. Within a thread, it is permissible to reorder reads of a final field with those modifications of a final field that do not take place in the constructor.

Appendix: On the bitwise manipulation

Essentially,

field.getModifiers() & ~Modifier.FINAL

turns off the bit corresponding to Modifier.FINAL from field.getModifiers(). & is the bitwise-and, and ~ is the bitwise-complement.

Remember Constant Expressions

Still not being able to solve this?, have fallen onto depression like I did for it? Does your code looks like this?

public class A {
    private final String myVar = "Some Value";
}

Reading the comments on this answer, specially the one by @Pshemo, it reminded me that Constant Expressions are handled different so it will be impossible to modify it. Hence you will need to change your code to look like this:

public class A {
    private final String myVar;

    private A() {
        myVar = "Some Value";
    }
}

if you are not the owner of the class... I feel you!

For more details about why this behavior read this?