Recently I have performed a fresh installation of Oracle Cloud Control 13.3. The first thing I have noticed was that the Weblogic admin server appears to be down according to Cloud Control.
When querying via emctl status oms -details everything seems ok:
[oracle@ora-cloud-control nodemanager]$ emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 3
Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : ora-cloud-control.localdomain
HTTP Console Port : 7788
HTTPS Console Port : 7803
HTTP Upload Port : 4889
HTTPS Upload Port : 4903
EM Instance Home : /u01/app/oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://ora-cloud-control.localdomain:7803/em
Upload URL: https://ora-cloud-control.localdomain:4903/empbs/upload
WLS Domain Information
Domain Name : GCDomain
Admin Server Host : ora-cloud-control.localdomain
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: ora-cloud-control.localdomain
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up
BI Publisher HTTP Managed Server Port : 9701
BI Publisher HTTPS Managed Server Port : 9803
BI Publisher HTTP OHS Port : 9788
BI Publisher HTTPS OHS Port : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://ora-cloud-control.localdomain:9851/xmlpserver/servlet/home
BI Publisher Server Logs: /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log : /u01/app/oracle/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log
However, the admin server cannot be reached at: https://ora-cloud-control.localdomain:7102/console nor via WLST:
wls:/offline> connect('weblogic','*******','t3s://ora-cloud-control.localdomain:7102')
Connecting to t3s://ora-cloud-control.localdomain:7102 with userid weblogic ...
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Apr 1, 2021 10:02:29 PM CEST> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>
Traceback (innermost last):
File "<console>", line 1, in ?
File "<iostream>", line 19, in connect
File "<iostream>", line 552, in raiseWLSTException
WLSTException: Error occurred while performing connect : Cannot connect via t3s or https. If using demo certs, verify that the -Dweblogic.security.TrustKeyStore=DemoTrust system property is set. : t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
Use dumpStack() to view the full stacktrace :
dumpStack() follows:
wls:/offline> dumpStack()
This Exception occurred at Thu Apr 01 22:02:30 CEST 2021.
javax.naming.CommunicationException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination [Root exception is java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:808)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:363)
at weblogic.jndi.Environment.getContext(Environment.java:319)
at weblogic.jndi.Environment.getContext(Environment.java:288)
at weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
at weblogic.management.scripting.WLSTHelper.populateInitialContext(WLSTHelper.java:519)
at weblogic.management.scripting.WLSTHelper.initDeprecatedConnection(WLSTHelper.java:570)
at weblogic.management.scripting.WLSTHelper.initConnections(WLSTHelper.java:310)
at weblogic.management.scripting.WLSTHelper.connect(WLSTHelper.java:200)
at weblogic.management.scripting.WLScriptContext.connect(WLScriptContext.java:67)
at weblogic.management.scripting.utils.WLSTUtil.initializeOnlineWLST(WLSTUtil.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.python.core.PyReflectedFunction.__call__(Unknown Source)
at org.python.core.PyMethod.__call__(Unknown Source)
at org.python.core.PyObject.__call__(Unknown Source)
at org.python.core.PyObject.invoke(Unknown Source)
at org.python.pycode._pyx7.connect$1(<iostream>:13)
at org.python.pycode._pyx7.call_function(<iostream>)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyFunction.__call__(Unknown Source)
at org.python.pycode._pyx87.f$0(<console>:1)
at org.python.pycode._pyx87.call_function(<console>)
at org.python.core.PyTableCode.call(Unknown Source)
at org.python.core.PyCode.call(Unknown Source)
at org.python.core.Py.runCode(Py.java:1226)
at org.python.core.Py.exec(Py.java:1252)
at org.python.util.PythonInterpreter.exec(Unknown Source)
at org.python.util.InteractiveInterpreter.runcode(Unknown Source)
at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
at org.python.util.InteractiveInterpreter.runsource(Unknown Source)
at weblogic.management.scripting.utils.WLSTInterpreter.runsource(WLSTInterpreter.java:910)
at weblogic.management.scripting.WLST.main(WLST.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at weblogic.WLST.main(WLST.java:29)
Caused by: java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:241)
at weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:169)
at weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:177)
at weblogic.jndi.WLInitialContextFactoryDelegate$1.run(WLInitialContextFactoryDelegate.java:342)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:337)
... 44 more
Caused by: java.rmi.ConnectException: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:489)
at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:327)
at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:309)
at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:213)
at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:263)
at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:225)
... 50 more
javax.naming.CommunicationException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination [Root exception is java.net.ConnectException: t3s://ora-cloud-control.localdomain:7102: Destination 127.0.0.1, 7102 unreachable; nested exception is:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem; No available router to destination]
wls:/offline>
I have tried three things to resolve the issue according to what I have found browsing the web:
-
appending
-Dweblogic.security.TrustKeyStore=DemoTrusttoJAVA_OPTIONSstring in/u01/app/oracle/gc_inst/user_projects/domains/GCDomain/bin/startWebLogic.shfile -
extracting the SSL certificate from
https://ora-cloud-control.localdomain:7803and importing it into the following keystores using this as a reference:/u01/app/oracle/middleware/oracle_common/jdk/jre/lib/security/cacerts /u01/app/oracle/middleware/wlserver/server/lib/cacerts /u01/app/oracle/agent/agent_13.3.0.0.0/oracle_common/jdk/jre/lib/security/cacerts -
importing the same certificate into
/u01/app/oracle/middleware/wlserver/server/lib/DemoTrust.jks trustore
I have tried to restart the whole OMS stack after each of the actions described above but with no luck, still getting the very same error message.
Any ideas please?
Best Answer
regarding connecting to admin sever locally via WLST:
I found out I was messing with the wrong certificate, e.g. that the SSL certificate generated during Cloud Control install and used for the https://ora-cloud-control.localdomain:7803/em site and the certificates used for Weblogic admin server were different.
Weblogic by default comes configured with two keystores - DemoIdentity.jks and DemoTrust.jks. More on this can be found here.
Issue in my case was that the demo private key that "came" with the installation was too short. This resulted in error present in the logs - "Received fatal alert certificate_unknown". I only noticed this later. See more on this topic here.
I have addressed this issue by regenerating DemoIdentity.jks and DemoTrust.jks according to following steps and also by adding the CertGenCA.der - the certificate of Certification Authority to JDK cacerts keystore (located in /u01/app/oracle/middleware/oracle_common/jdk/jre/lib/security/cacerts in my case).
I could finally connect to admin server via wlst.sh.
However, I still haven't been able to connect to weblogic remotely. I found out that weblogic has been listening on 172.0.0.1:7102 via netstat. I have tried to change its listen address via $WL_DOMAIN/config/config.xml but without success.
Finally I found this article stating: WebLogic Server listens on all host names that are associated with the hosting machine—namely, both the machine host name and localhost and realized my /etc/hosts missed the local IP address and contained "127.0.0.1" only. After fixing this and restarting the OMS stack, the admin server began listening on 192.168.0.50:7102 and could be thus accessed remotely.
Hope this helps someone. Regards, Michal