Tuesday, January 7, 2014

The object identified by: '31' could not be found

The Problem Abstract 

I encountered a strange exception while restarting my admin WebLogic server. Everything was working fine a day ago. But a day later its was giving me strange issue of "object 31" error.
After struggling for a while I realized that some startup option was creating the issue. The startup options was containing JVM arguments "-Dweblogic.management.server=t3://10.188.103.55:7002". The admin server should not contain this, as it makes the server to run in managed server mode. So the issue was, admin server was starting as managed server and trying to contact itself, creating a deadlock kind scenario.

Usually ${ADMIN_URL} sets up the variable "weblogic.management.server". I tried to find out that who set this variable for admin mode startup, as setDomain.sh script enforces this variable is null for admin server startup sequence. On little debugging on environment variables at operating system level, it was clear ${JAVA_OPTS} was exported in the UNIX session, as one can also see in the startup sequence that the variables are repeated. So the sequence of events which landed me in this soup was:



  1. I started one managed server which exported JAVA_OPTS variable in the system session.
  2. When I restarted admin server, it appended its options in already exported JAVA_OPTS as the variable was not cleared after managed server start/ restart.

Solution 

  1. Either manually purge the startup variables from UNIX, Linux session if you are stubborn and don't want to close your session.
  2. Kill the UNIX Linux session. Start a new session, it will automatically purge the exported variables. Now start your admin server. 

Example Log excerpt with startup options 

-Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/apps2/BEA/WebLogic1036/wlserver_10.3/server/lib/cacerts -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/apps2/BEA/WebLogic1036/wlserver_10.3/server/lib/cacerts -Dweblogic.ProductionModeEnabled=true -Dweblogic.ProductionModeEnabled=true -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=http://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic2

<<WLS Kernel>> <> <> <1389043279519> <BEA-400000> <Initializing message interception service>
####<Jan 6, 2014 1:21:19 PM PST> <Error> <Configuration Management> <dh6802c1> <OWSAdmin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1389043279801> <BEA-150000> <An error occurred while establishing a connection back to the adminstration server t3://141.204.67.218:7002 during startup. Since bootstrap succeeded check that t3://141.204.67.218:7002 uniquely identifies the administration server.
javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:455)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<Jan 6, 2014 1:21:19 PM PST> <Critical> <WebLogicServer> <dh6802c1> <OWSAdmin> <main> <<WLS Kernel>> <> <> <1389043279816> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:455)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        ... 9 more