[LPS-80843] Clean up DS components soft circular dependencies to prevent potential concurrent bundles starting deadlocks and create a defense mechanism to prevent new soft circular dependencies from being added in the future. - Liferay Issues

XML

Word

Printable

Details

Type: Bug
Status: Closed
Resolution: Fixed
Affects Version/s: Master
Fix Version/s: 7.1.0 Beta 1, 7.1.X, Master
Component/s: Core Infrastructure, Portal Services > Component Management > Module Framework (OSGi)
Labels:
None

Git Pull Request:
https://github.com/brianchandotcom/liferay-portal/pull/58362

Description

"Soft circular dependencies" means a circular dependencies with at least one non-mandatory (aka, optional or multiple) references.

This kind of circles are not a problem for SCR.

However when components in the circle are from different bundles, and the bundles are started concurrently from different threads, there is some chance we run into deadlock within osgi core.
Here is an example:

     [exec] Found one Java-level deadlock:
     [exec] =============================
     [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
     [exec]   waiting for ownable synchronizer 0x000000008cb02f18, (a java.util.concurrent.locks.ReentrantLock$FairSync),
     [exec]   which is held by "main"
     [exec] "main":
     [exec]   waiting to lock monitor 0x00007fc708005c28 (object 0x000000008cc0dbc8, a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse),
     [exec]   which is held by "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules"
     [exec] 
     [exec] Java stack information for the threads listed above:
     [exec] ===================================================
     [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
     [exec] 	at sun.misc.Unsafe.park(Native Method)
     [exec] 	- parking to wait for  <0x000000008cb02f18> (a java.util.concurrent.locks.ReentrantLock$FairSync)
     [exec] 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
     [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
     [exec] 	at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:227)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainStateLock(AbstractComponentManager.java:278)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:806)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0d9b8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
     [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
     [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
     [exec] 	- locked <0x000000008cb508f8> (a java.lang.Object)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
     [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
     [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
     [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
     [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
     [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
     [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1258)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1230)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:512)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:361)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:312)
     [exec] "main":
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:490)
     [exec] 	- waiting to lock <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0c228> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1714)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.open(DependencyManager.java:1556)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:260)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:108)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:906)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:879)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0cdb8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
     [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
     [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
     [exec] 	- locked <0x000000008cb1e740> (a java.lang.Object)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
     [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
     [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
     [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
     [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
     [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
     [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:421)
     [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl._startDynamicBundles(ModuleFrameworkImpl.java:1566)
     [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl.startFramework(ModuleFrameworkImpl.java:412)
     [exec] 	at com.liferay.portal.module.framework.ModuleFrameworkUtilAdapter.startFramework(ModuleFrameworkUtilAdapter.java:99)
     [exec] 	at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:257)
     [exec] 	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4579)
     [exec] 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5041)
     [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
     [exec] 	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:742)
     [exec] 	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:718)
     [exec] 	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630)
     [exec] 	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1840)
     [exec] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
     [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:525)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:424)
     [exec] 	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1586)
     [exec] 	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:308)
     [exec] 	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
     [exec] 	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:966)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:839)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1427)
     [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1417)
     [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
     [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
     [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:258)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000821616e8> (a org.apache.catalina.core.StandardService)
     [exec] 	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:770)
     [exec] 	- locked <0x00000000821619d0> (a java.lang.Object)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x0000000082161670> (a org.apache.catalina.core.StandardServer)
     [exec] 	at org.apache.catalina.startup.Catalina.start(Catalina.java:682)
     [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     [exec] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [exec] 	at java.lang.reflect.Method.invoke(Method.java:498)
     [exec] 	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
     [exec] 	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
     [exec] 
     [exec] Found 1 deadlock.

This happened because main thread is starting com.liferay.portal.messaging bundle while osgi/modules FileInstall thread is starting com.liferay.adaptive.media.web bundle at the same time.

And we had this soft circular dependencies:

com.liferay.portal.messaging.internal.DefaultMessageBus[registerMessageListener] -> 
		 com.liferay.adaptive.media.web.internal.messaging.AMMessageListener[_amAsyncProcessorLocator] -> 
			com.liferay.adaptive.media.web.internal.processor.AMAsyncProcessorLocatorImpl[_messageBus] -> (circular reference)

Although it is technically ok to have soft circular dependencies from SCR's point of view. But due the concurrent nature in our system, we want to eliminate the possibility of seeing deadlock during boot up and runtime bundles deployment.

To get to that goal, the following things have be done in this ticket:

During the boot up time, we force main thread to be the only one starting all bundles, therefore eliminate the concurrency here. So we won't have random startup time deadlock anymore, if there is any deadlock due to some future breaking changes, it will be a stable/reproducible deadlock, not random one. This way it is easy to see and fix the problem.
Create a scanner (it contains system checker and gogo shell command) for soft circular dependencies scanning. So from now on, developers will see their soft circular dependencies report on boot up, and they can also use the gogo shell command to rerun it. And CI will be catching and failing on new soft circular dependencies.
All exist soft circular dependencies reported by 2) from our current codebase are fixed.

Some limitation note:
In order to scan faster, components whom are involved in multiple circles (because component could have multiple @Reference fields/methods which could lead to different circles), only the first circle is reported. The scan is navigated through a determinate sorted order, which means for the same graph we always report the same "first" circles for components involve multiple circles. This is important for generating a stable and reproducible report across all systems.

Attachments

Issue Links

relates

LRDOCS-5324 Report soft circular dependencies using the system:check Gogo shell command

Reopened

Activity

People

Assignee:

Brian Chan

Reporter:

Shuyang Zhou

Participants of an Issue:

Brian Chan, Jason Pince, Shuyang Zhou

Recent user:

Michael Saechang

Votes:

0 Vote for this issue

Watchers:

1 Start watching this issue

Dates

Created:

12/May/18 12:25 AM

Updated:

07/Sep/18 10:43 AM

Resolved:

14/May/18 4:44 PM

Days since last comment:

2 years, 12 weeks, 5 days ago

Packages

Version	Package
7.1.0 Beta 1
7.1.X
Master