Java – Can’t connect to weblogic server remotely using JConsole

I need to monitor a weblogic server running on a remote machine using JConsole. Just for now, while I'm trying I'm testing everything on my machine, that is I'm trying to make it work remotely on my machine, when this works I will make the necessary changes on the server but as the application on the server is being used for other users I'm not able to do the changes and stop/start the server very often. So, I'm using my pc as the server and to connect "remotely" using JConsole.

Environment: both, my pc and the server are running with WL 10.3; the server is running with Java Sun JDK 1.6.0_27 and Solaris OS, and my pc is running with Java Sun JDK 1.6.0_26 and Ubuntu OS.

I'm able to connect Locally, running JConsole and clicking on the weblogic process.

The application deployment is pretty big, it runs a few scripts. When the domain of weblogic is created I set the following properties:

java -classpath ${CLASSPATH} ${MY_OPTS} -Dweblogic.management.username=${ADMIN_USER}  \
    -Dweblogic.management.password=${ADMIN_PASSWD} -Dweblogic.Domain=${MY_DOMAIN} \
    -Dweblogic.Name=${MY_SERVER} -Dweblogic.ListenPort=${MY_PORT} \
    -Dweblogic.RootDirectory=${MY_DOMAIN_HOME}/${MY_DOMAIN} \
    -Dcom.sun.management.jmxremote.port=22222 \
    -Dcom.sun.management.jmxremote.authenticate=true \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.password.file=/home/MY_USER/jmxremote.password \
    -Dcom.sun.management.jmxremote.access.file=/home/MY_USER/jmxremote.access \
    -Dweblogic.management.GenerateDefaultConfig=true weblogic.Server &

I also add at some point this property -Djava.rmi.server.hostname= –> I'm not using neither localhost nor 127.0.0.1, but it made no difference

Then, I run jconsole -debug from console and try to connect with the remote option using the following service url (I try different one):

service:jmx:rmi:///jndi/rmi://127.0.0.1:22222/jmxrmi
service:jmx:rmi:///jndi/rmi://<my_ip>:22222/jmxrmi
service:jmx:rmi://127.0.0.1:22222/jndi/rmi://127.0.0.1:1099/

1099 is the port where rmiregistry runs, but I have to start it manually with rmiregistry &, otherwise that port is not listening.

While the application is running I run netstat command, though I see ports 8080, 8081 (SSL port) and the port of my administration console (which it is not the default 7001) I don't see the port 22222 listening, and I think I should, right?

I've even add the following at weblogic.policy:

grant {
permission java.net.SocketPermission
"<my_ip>:1024-65535", "connect,resolve";
};

In the console while the application is deploying I see:

Connector Server Address = service:jmx:iiop://<my_ip>:8080    /jndi/weblogic.management.mbeanservers.runtime

Connector Server Address = service:jmx:iiop://:8080/jndi/weblogic.management.mbeanservers.edit

Connector Server Address = service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc0AAtVbmljYXN0UmVmMgAACTEyNy4wLjEuMQAAxMelHYZki5P74FdoVWYAAAEzsm1U1oACAHg=

Conector JMX ready in: service:jmx:rmi:///jndi/rmi://usuario-System-Product-Name:22222/jmxrmi

I try these ones too but unsuccessfully.

The only way I'm able to connect is if I first access locally using the pid, then in console I get a service url similar to the one I get in the logs:

service:jmx:rmi://127.0.0.1/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp/tzJi+FlGgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc0AAtVbmljYXN0UmVmMgAACTEyNy4wLjEuMQAAgDBSpbjGvIybrBekgUYAAAEzsm9/YIASAHg=

Using this url in the remote section I'm able to connect, it is similar but not the same to the one logged, which btw doesn't work.

The errors I get in JConsole console when using -debug are:

Some of the errors I got in the console are, they are not random.. I mean, different errors while I tried different urls, running /not running rmiregistry manually, etc.:

Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file:  java.naming.factory.initial

11/11/2011 09:46:46 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://127.0.0.1:22222/jmxrmi] connecting...
11/11/2011 09:46:46 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://127.0.0.1:22222/jmxrmi] finding stub...
11/11/2011 09:46:46 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://127.0.0.1:22222/jmxrmi] Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: 
non-JRMP server at remote endpoint]

Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: non-JRMP server at remote endpoint]

17/11/2011 12:08:28 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://<my_ip>:22222/jmxrmi] connecting...
17/11/2011 12:08:28 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://<my_ip>:22222/jmxrmi] finding stub...
17/11/2011 12:08:28 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://<my_ip>:22222/jmxrmi] Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: jmxrmi
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:340)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
    at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:350)
    at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:296)
    at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:280)
Caused by: javax.naming.NameNotFoundException: jmxrmi
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:99)
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1888)
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1858)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
    ... 4 more

Notes: When I used I'm meaning 172.xx.xxx.xxx

What am I doing wrong? What else should I set / check / change?

Well I've a few changes but I still can't make it worked.

I set the

java -classpath ${CLASSPATH} ${MY_OPTS} -Dweblogic.management.username=${ADMIN_USER} \ -Dweblogic.management.password=${ADMIN_PASSWD} -Dweblogic.Domain=${MY_DOMAIN} \ -Dweblogic.Name=${MY_SERVER} -Dweblogic.ListenPort=${MY_PORT} \ -Dweblogic.RootDirectory=${MY_DOMAIN_HOME}/${MY_DOMAIN} \ -Djava.rmi.server.hostname=${ADMIN_HOST} \ -Dcom.sun.management.jmxremote.port=22222 \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.password.file=/home/MY_USER/jmxremote.password \ -Dcom.sun.management.jmxremote.access.file=/home/MY_USER/jmxremote.access \ -Dweblogic.management.GenerateDefaultConfig=true weblogic.Server &

I changed the authentication property to false and added the java.rmi.server.hostname where ADMIN_HOST is a variable set to the ip 172.17.209.66 of my computer.

When I used the service url in JConsole: service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi

I get the following exception

22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] connecting...
22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] finding stub...
22/11/2011 05:22:54 RMIConnector connect
MÁS FINA: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://172.17.209.66:22222/jmxrmi] Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 172.17.209.66; nested exception is: 
    java.net.ConnectException: Conexión rehusada]
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 172.17.209.65; nested exception is: 
    java.net.ConnectException: Conexión rehusada]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:340)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)

And in the logs of my application I see the following:

Conector JMX ready in: service:jmx:rmi:///jndi/rmi://usuario-System-Product-Name:22222/jmxrmi

The /etc/hosts shows:

172.17.209.65 usuario-System-Product-Name # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost ::1 usuario-System-Product-Name localhost6.localdomain6 localhost6
127.0.1.1 usuario-System-Product-Name

is that why I see the usuario-System-Product-Name and not the ip? Even if I set as hostname the ip of my computer?

I disabled both firewals and try the command line jmx tool Alex suggested buy with no luck 🙁

Any other suggestions / tips / help ?