Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-80843

Clean up DS components soft circular dependencies to prevent potential concurrent bundles starting deadlocks and create a defense mechanism to prevent new soft circular dependencies from being added in the future.

    Details

      Description

      "Soft circular dependencies" means a circular dependencies with at least one non-mandatory (aka, optional or multiple) references.

      This kind of circles are not a problem for SCR.

      However when components in the circle are from different bundles, and the bundles are started concurrently from different threads, there is some chance we run into deadlock within osgi core.
      Here is an example:

           [exec] Found one Java-level deadlock:
           [exec] =============================
           [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
           [exec]   waiting for ownable synchronizer 0x000000008cb02f18, (a java.util.concurrent.locks.ReentrantLock$FairSync),
           [exec]   which is held by "main"
           [exec] "main":
           [exec]   waiting to lock monitor 0x00007fc708005c28 (object 0x000000008cc0dbc8, a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse),
           [exec]   which is held by "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules"
           [exec] 
           [exec] Java stack information for the threads listed above:
           [exec] ===================================================
           [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
           [exec] 	at sun.misc.Unsafe.park(Native Method)
           [exec] 	- parking to wait for  <0x000000008cb02f18> (a java.util.concurrent.locks.ReentrantLock$FairSync)
           [exec] 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
           [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
           [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
           [exec] 	at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:227)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainStateLock(AbstractComponentManager.java:278)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:806)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
           [exec] 	at java.security.AccessController.doPrivileged(Native Method)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
           [exec] 	- locked <0x000000008cc0d9b8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
           [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
           [exec] 	at java.security.AccessController.doPrivileged(Native Method)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
           [exec] 	- locked <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
           [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
           [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
           [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
           [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
           [exec] 	- locked <0x000000008cb508f8> (a java.lang.Object)
           [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
           [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
           [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
           [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
           [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
           [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
           [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
           [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
           [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
           [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1258)
           [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1230)
           [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:512)
           [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:361)
           [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:312)
           [exec] "main":
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:490)
           [exec] 	- waiting to lock <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
           [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
           [exec] 	at java.security.AccessController.doPrivileged(Native Method)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
           [exec] 	- locked <0x000000008cc0c228> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
           [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1714)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.open(DependencyManager.java:1556)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:260)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:108)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:906)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:879)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
           [exec] 	at java.security.AccessController.doPrivileged(Native Method)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
           [exec] 	- locked <0x000000008cc0cdb8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
           [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
           [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
           [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
           [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
           [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
           [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
           [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
           [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
           [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
           [exec] 	- locked <0x000000008cb1e740> (a java.lang.Object)
           [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
           [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
           [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
           [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
           [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
           [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
           [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
           [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
           [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
           [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
           [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
           [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:421)
           [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl._startDynamicBundles(ModuleFrameworkImpl.java:1566)
           [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl.startFramework(ModuleFrameworkImpl.java:412)
           [exec] 	at com.liferay.portal.module.framework.ModuleFrameworkUtilAdapter.startFramework(ModuleFrameworkUtilAdapter.java:99)
           [exec] 	at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:257)
           [exec] 	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4579)
           [exec] 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5041)
           [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
           [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
           [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
           [exec] 	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:742)
           [exec] 	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:718)
           [exec] 	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
           [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630)
           [exec] 	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1840)
           [exec] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
           [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
           [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:525)
           [exec] 	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:424)
           [exec] 	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1586)
           [exec] 	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:308)
           [exec] 	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
           [exec] 	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
           [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
           [exec] 	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
           [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
           [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:966)
           [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
           [exec] 	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:839)
           [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
           [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
           [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
           [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1427)
           [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1417)
           [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
           [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
           [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
           [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
           [exec] 	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:258)
           [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
           [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
           [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
           [exec] 	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
           [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
           [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
           [exec] 	- locked <0x00000000821616e8> (a org.apache.catalina.core.StandardService)
           [exec] 	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:770)
           [exec] 	- locked <0x00000000821619d0> (a java.lang.Object)
           [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
           [exec] 	- locked <0x0000000082161670> (a org.apache.catalina.core.StandardServer)
           [exec] 	at org.apache.catalina.startup.Catalina.start(Catalina.java:682)
           [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           [exec] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           [exec] 	at java.lang.reflect.Method.invoke(Method.java:498)
           [exec] 	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
           [exec] 	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
           [exec] 
           [exec] Found 1 deadlock.
      

      This happened because main thread is starting com.liferay.portal.messaging bundle while osgi/modules FileInstall thread is starting com.liferay.adaptive.media.web bundle at the same time.

      And we had this soft circular dependencies:

      com.liferay.portal.messaging.internal.DefaultMessageBus[registerMessageListener] -> 
      		 com.liferay.adaptive.media.web.internal.messaging.AMMessageListener[_amAsyncProcessorLocator] -> 
      			com.liferay.adaptive.media.web.internal.processor.AMAsyncProcessorLocatorImpl[_messageBus] -> (circular reference)
      

      Although it is technically ok to have soft circular dependencies from SCR's point of view. But due the concurrent nature in our system, we want to eliminate the possibility of seeing deadlock during boot up and runtime bundles deployment.

      To get to that goal, the following things have be done in this ticket:

      1. During the boot up time, we force main thread to be the only one starting all bundles, therefore eliminate the concurrency here. So we won't have random startup time deadlock anymore, if there is any deadlock due to some future breaking changes, it will be a stable/reproducible deadlock, not random one. This way it is easy to see and fix the problem.
      2. Create a scanner (it contains system checker and gogo shell command) for soft circular dependencies scanning. So from now on, developers will see their soft circular dependencies report on boot up, and they can also use the gogo shell command to rerun it. And CI will be catching and failing on new soft circular dependencies.
      3. All exist soft circular dependencies reported by 2) from our current codebase are fixed.

      Some limitation note:
      In order to scan faster, components whom are involved in multiple circles (because component could have multiple @Reference fields/methods which could lead to different circles), only the first circle is reported. The scan is navigated through a determinate sorted order, which means for the same graph we always report the same "first" circles for components involve multiple circles. This is important for generating a stable and reproducible report across all systems.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  48 weeks, 4 days ago