How to Fix ‘Failed to Retrieve RMIServer Stub’ JMX Error in Tomcat

javajmxtomcat

I just completed a migration from Windows server 2008 R2 to Ubuntu 10.04. I manage a java application (Java 6, Tomcat) that is having some performance issues. I'd like to use JMX to try and troubleshoot, but I can't seem to get jvisualvm to connect.

If I do a ps -ef | grep "java", I see the following parameters.

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

Netstat shows that port 8084 is listening on 0.0.0.0. In my config, JMX is setup to bind to the FQDN of the server (we use a private DNS server). My firewall (IPTABLES/UFW) is setup to allow all outgoing traffic, and to allow incoming traffic on port 8084.

The server itself is virtual with two NICs, a public and a private. The public NIC's gateway is disabled so that connections can only come in on the private side.

When I try to connect jvisualvm to my app server using JMX, I get the following error in jvisualvm.

Cannot connect to [FQDN OMITTED]:8084 using server:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi

If I look in the jvisualvm log, I see the following trace.

NFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi)
java.io.EOFException: SSL peer shut down incorrectly
                at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
                at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                at java.io.DataOutputStream.flush(DataOutputStream.java:106)
                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
                at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
                at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
                at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
                at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
                at javax.naming.InitialContext.lookup(InitialContext.java:392)
                at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
                at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:451)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:395)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:216)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:205)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
                at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
                at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:267)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:185)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:283)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:261)
                at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
                at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:577)
                at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1030)

Does anyone have any ideas?

Best Answer

You can specify same port for both JMX and RMI conversation:

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.rmi.port=8084