Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-102853

Inconsistent failure to stop threads on shutdown

    Description

    Steps to Reproduce:

    1. Start server
    2. Shutdown server

    Result (inconsistently):

    04-Oct-2019 08:18:10.322 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Framework Event Dispatcher: Equinox Container: 71a57982-cf41-4569-8307-561c8232106e] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.lang.Object.wait(Object.java:502)
     org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
     org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
    04-Oct-2019 08:18:10.323 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Refresh Thread: Equinox Container: 71a57982-cf41-4569-8307-561c8232106e] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.lang.Object.wait(Object.java:502)
     org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
     org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
    04-Oct-2019 08:18:10.323 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Framework stop] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1590)
     org.eclipse.osgi.container.SystemModule.stopWorker(SystemModule.java:270)
     org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule.stopWorker(EquinoxBundle.java:192)
     org.eclipse.osgi.container.Module.doStop(Module.java:636)
     org.eclipse.osgi.container.Module.stop(Module.java:498)
     org.eclipse.osgi.container.SystemModule.stop(SystemModule.java:202)
     org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule$1.run(EquinoxBundle.java:210)
     java.lang.Thread.run(Thread.java:745)
    

    Side effect:
    When attempting to restart the portal, module framework console port still bound:

    java.lang.Exception: Port 11312 already in use
    	at org.eclipse.equinox.console.telnet.TelnetCommand.telnet(TelnetCommand.java:164)
    	at org.eclipse.equinox.console.telnet.TelnetCommand.startService(TelnetCommand.java:101)
    	at org.eclipse.equinox.console.command.adapter.Activator$ProcessorCustomizer.addingService(Activator.java:87)
    	at org.eclipse.equinox.console.command.adapter.Activator$ProcessorCustomizer.addingService(Activator.java:1)
    	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:943)
    	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:1)
    	at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
    	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229)
    	at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:903)
    	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
    	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:891)
    	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
    	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
    	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:469)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:487)
    	at org.apache.felix.gogo.runtime.activator.Activator.newProcessor(Activator.java:74)
    	at org.apache.felix.gogo.runtime.activator.Activator.start(Activator.java:83)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:779)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:772)
    	at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:729)
    	at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1002)
    	at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:354)
    	at org.eclipse.osgi.container.Module.doStart(Module.java:581)
    	at org.eclipse.osgi.container.Module.start(Module.java:449)
    	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:428)
    	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:447)
    	at com.liferay.portal.bootstrap.ModuleFrameworkImpl.lambda$_setUpInitialBundles$2(ModuleFrameworkImpl.java:1711)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    

    Tests are setting property:

    module.framework.properties.osgi.console=11312
    

    Only seen on CI so far, and the failure is not consistent.

    Similar issue seen on SP1 bundle (LPS-102837), but the issue on the SP1 bundle is consistent and does not cause issues on restart.

    Failure examples:
    https://testray.liferay.com/reports/production/logs/test-1-1/1570235591300/test-portal-acceptance-upstream(master)/6761/functional-tomcat90-mysql57-jdk8/58/0/jenkins-console.txt.gz
    https://testray.liferay.com/reports/production/logs/test-1-1/1570235591300/test-portal-acceptance-upstream(master)/6761/functional-tomcat90-sybase160-jdk8/3/0/jenkins-console.txt.gz
    https://testray.liferay.com/reports/production/logs/test-1-1/1570172613982/test-portal-acceptance-upstream(7.2.x)/723/lpkg-persistence-jdk8/0/jenkins-console.txt.gz

      Attachments

        Issue Links

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                14 weeks, 6 days ago

                Packages

                Version Package
                7.2.10 DXP FP4
                7.2.X
                7.3.0 CE GA1
                7.3.1 CE GA2
                Master