Clean up DS components soft circular dependencies to prevent potential concurrent bundles starting deadlocks and create a defense mechanism to prevent new soft circular dependencies from being added in the future.

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: Master
• Fix Version/s: 7.1.0 Beta 1, 7.1.X, Master
• Component/s: Core Infrastructure, Portal Services > Component Management > Module Framework (OSGi)
• Labels: None
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/58362

"Soft circular dependencies" means a circular dependencies with at least one non-mandatory (aka, optional or multiple) references.

This kind of circles are not a problem for SCR.

However when components in the circle are from different bundles, and the bundles are started concurrently from different threads, there is some chance we run into deadlock within osgi core.
Here is an example:

     [exec] Found one Java-level deadlock:
     [exec] =============================
     [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
     [exec]   waiting for ownable synchronizer 0x000000008cb02f18, (a java.util.concurrent.locks.ReentrantLock$FairSync),
     [exec]   which is held by "main"
     [exec] "main":
     [exec]   waiting to lock monitor 0x00007fc708005c28 (object 0x000000008cc0dbc8, a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse),
     [exec]   which is held by "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules"
     [exec] 
     [exec] Java stack information for the threads listed above:
     [exec] ===================================================
     [exec] "fileinstall-/opt/dev/projects/github/liferay-portal/bundles/osgi/modules":
     [exec] 	at sun.misc.Unsafe.park(Native Method)
     [exec] 	- parking to wait for  <0x000000008cb02f18> (a java.util.concurrent.locks.ReentrantLock$FairSync)
     [exec] 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
     [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
     [exec] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
     [exec] 	at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:227)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainStateLock(AbstractComponentManager.java:278)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:806)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0d9b8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
     [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
     [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
     [exec] 	- locked <0x000000008cb508f8> (a java.lang.Object)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
     [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
     [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
     [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
     [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
     [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
     [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1258)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1230)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:512)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:361)
     [exec] 	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:312)
     [exec] "main":
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:490)
     [exec] 	- waiting to lock <0x000000008cc0dbc8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.FieldHandler$ReferenceMethodImpl.getServiceObject(FieldHandler.java:985)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0c228> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1714)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.open(DependencyManager.java:1556)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:260)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:108)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:906)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:879)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:823)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:212)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:210)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:111)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:45)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:499)
     [exec] 	- locked <0x000000008cc0cdb8> (a org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:461)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:619)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72)
     [exec] 	at org.apache.felix.scr.impl.inject.BindMethod.getServiceObject(BindMethod.java:648)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2201)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1118)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1520)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1005)
     [exec] 	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:859)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
     [exec] 	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
     [exec] 	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
     [exec] 	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
     [exec] 	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
     [exec] 	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:429)
     [exec] 	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:657)
     [exec] 	- locked <0x000000008cb1e740> (a java.lang.Object)
     [exec] 	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:341)
     [exec] 	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:403)
     [exec] 	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:54)
     [exec] 	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:278)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259)
     [exec] 	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
     [exec] 	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232)
     [exec] 	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444)
     [exec] 	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
     [exec] 	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
     [exec] 	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
     [exec] 	at org.eclipse.osgi.container.Module.start(Module.java:452)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402)
     [exec] 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:421)
     [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl._startDynamicBundles(ModuleFrameworkImpl.java:1566)
     [exec] 	at com.liferay.portal.bootstrap.ModuleFrameworkImpl.startFramework(ModuleFrameworkImpl.java:412)
     [exec] 	at com.liferay.portal.module.framework.ModuleFrameworkUtilAdapter.startFramework(ModuleFrameworkUtilAdapter.java:99)
     [exec] 	at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:257)
     [exec] 	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4579)
     [exec] 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5041)
     [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x0000000082a78810> (a org.apache.catalina.core.StandardContext)
     [exec] 	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:742)
     [exec] 	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:718)
     [exec] 	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:703)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:630)
     [exec] 	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1840)
     [exec] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
     [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:525)
     [exec] 	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:424)
     [exec] 	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1586)
     [exec] 	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:308)
     [exec] 	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
     [exec] 	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:966)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:839)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000829cdca8> (a org.apache.catalina.core.StandardHost)
     [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1427)
     [exec] 	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1417)
     [exec] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     [exec] 	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
     [exec] 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
     [exec] 	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:943)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:258)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
     [exec] 	- locked <0x00000000829cd608> (a org.apache.catalina.core.StandardEngine)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x00000000821616e8> (a org.apache.catalina.core.StandardService)
     [exec] 	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:770)
     [exec] 	- locked <0x00000000821619d0> (a java.lang.Object)
     [exec] 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
     [exec] 	- locked <0x0000000082161670> (a org.apache.catalina.core.StandardServer)
     [exec] 	at org.apache.catalina.startup.Catalina.start(Catalina.java:682)
     [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [exec] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     [exec] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [exec] 	at java.lang.reflect.Method.invoke(Method.java:498)
     [exec] 	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
     [exec] 	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
     [exec] 
     [exec] Found 1 deadlock.

This happened because main thread is starting com.liferay.portal.messaging bundle while osgi/modules FileInstall thread is starting com.liferay.adaptive.media.web bundle at the same time.

And we had this soft circular dependencies:

com.liferay.portal.messaging.internal.DefaultMessageBus[registerMessageListener] -> 
		 com.liferay.adaptive.media.web.internal.messaging.AMMessageListener[_amAsyncProcessorLocator] -> 
			com.liferay.adaptive.media.web.internal.processor.AMAsyncProcessorLocatorImpl[_messageBus] -> (circular reference)

Although it is technically ok to have soft circular dependencies from SCR's point of view. But due the concurrent nature in our system, we want to eliminate the possibility of seeing deadlock during boot up and runtime bundles deployment.

To get to that goal, the following things have be done in this ticket:

1. During the boot up time, we force main thread to be the only one starting all bundles, therefore eliminate the concurrency here. So we won't have random startup time deadlock anymore, if there is any deadlock due to some future breaking changes, it will be a stable/reproducible deadlock, not random one. This way it is easy to see and fix the problem.
2. Create a scanner (it contains system checker and gogo shell command) for soft circular dependencies scanning. So from now on, developers will see their soft circular dependencies report on boot up, and they can also use the gogo shell command to rerun it. And CI will be catching and failing on new soft circular dependencies.
3. All exist soft circular dependencies reported by 2) from our current codebase are fixed.

Some limitation note:
In order to scan faster, components whom are involved in multiple circles (because component could have multiple @Reference fields/methods which could lead to different circles), only the first circle is reported. The scan is navigated through a determinate sorted order, which means for the same graph we always report the same "first" circles for components involve multiple circles. This is important for generating a stable and reproducible report across all systems.