Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-69494

Override Felix AbstractExtender to fix the concurrent bundle stopping race condition

    Details

      Description

      org.apache.felix.utils.extender.AbstractExtender has a broken design when dealing with concurrent bundle stopping.

      In a scenario like this, an AbstractExtender has started with Bundle A's BundleContext, and another Bundle B has started.

      1. Bundle B stops, because AbstractExtender registered itself as SynchronousBundleListener when started, so its bundleChanged(BundleEvent) method is invoked with Bundle B's STOPPING event.
      2. Inside AbstractExtender.bundleChanged(BundleEvent), there is a call to context.getBundle(), the "context" is Bundle A's BundleContext, because AbstractExtender was started with it. In order to be able to do this "context.getBundle()", the context must be valid, which means Bundle A has not been STOPPED yet. Now let's assume the current notifiying thread pauses right before this call. (Maybe due to CPU scheduling, context switching, system slowness or plainly debugger break points)
      3. Now another thread stops Bundle A (Maybe from gogo or just a concurrent system shutdown), and it proceeds successfully. So Bundle A is STOPPED, and its BundleContext is no longer valid.
      4. Now let 2)'s notifying thread resume to proceed to "context.getBundle()", now this context is no longer valid, therefore we get a "java.lang.IllegalStateException: BundleContext is no longer valid"

      This kind of error has happened many times on CI, although it is still very rare. It can only happen to a normal production system during a concurrent bundle stopping with certain sequence, which is not easy to replicate, but it can be very easily replicated with the above step by using gogo and debugger.
      The error stack can be very vary, due to some AbstractExtender's Extenstions use the owner's BundleContext(like the 2) case below, which is ThemeContributorExtender using org.apache.felix.utils.log.Logger, which in turn uses the owner BundleContext), the race condition can kick in before any BundleContext access casuing this IllegalStateException.

      Here are 2 sample error stack from CI:
      1)

      java.lang.IllegalStateException: BundleContext is no longer valid
           	at org.eclipse.osgi.internal.framework.BundleContextImpl.checkValid(BundleContextImpl.java:986)
           	at org.eclipse.osgi.internal.framework.BundleContextImpl.ungetService(BundleContextImpl.java:659)
           	at org.osgi.util.tracker.ServiceTracker.removedService(ServiceTracker.java:459)
           	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:967)
           	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1)
           	at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
           	at org.osgi.util.tracker.ServiceTracker.close(ServiceTracker.java:377)
           	at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:549)
           	at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:414)
           	at org.apache.felix.scr.impl.Activator.access$300(Activator.java:53)
           	at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:273)
           	at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
           	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           	at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
           	at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
           	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
           	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
           	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
           	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
           	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
           	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
           	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
           	at org.eclipse.osgi.container.Module.stop(Module.java:496)
           	at org.eclipse.osgi.container.ModuleContainer.unresolve0(ModuleContainer.java:841)
           	at org.eclipse.osgi.container.ModuleContainer.unresolve(ModuleContainer.java:755)
           	at org.eclipse.osgi.container.ModuleContainer.refresh(ModuleContainer.java:953)
           	at org.eclipse.osgi.container.ModuleContainer$ContainerWiring.dispatchEvent(ModuleContainer.java:1336)
           	at org.eclipse.osgi.container.ModuleContainer$ContainerWiring.dispatchEvent(ModuleContainer.java:1)
           	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
           	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)
      

      2)

      java.lang.IllegalStateException: BundleContext is no longer valid
          	at org.eclipse.osgi.internal.framework.BundleContextImpl.checkValid(BundleContextImpl.java:986)
          	at org.eclipse.osgi.internal.framework.BundleContextImpl.ungetService(BundleContextImpl.java:659)
          	at org.apache.felix.utils.log.Logger._log(Logger.java:104)
          	at org.apache.felix.utils.log.Logger.log(Logger.java:77)
          	at org.apache.felix.utils.log.Logger.log(Logger.java:69)
          	at com.liferay.frontend.theme.contributor.extender.internal.ThemeContributorExtender.debug(ThemeContributorExtender.java:67)
          	at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:303)
          	at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
          	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:905)
          	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
          	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
          	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:165)
          	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:75)
          	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:67)
          	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:102)
          	at org.eclipse.osgi.container.Module.publishEvent(Module.java:461)
          	at org.eclipse.osgi.container.Module.doStop(Module.java:619)
          	at org.eclipse.osgi.container.Module.stop(Module.java:483)
          	at org.eclipse.osgi.container.SystemModule.stop(SystemModule.java:186)
          	at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule$1.run(EquinoxBundle.java:159)
          	at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              brian.chan Brian Chan
              Reporter:
              shuyang.zhou Shuyang Zhou
              Participants of an Issue:
              Recent user:
              Csaba Turcsan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                2 years, 49 weeks, 5 days ago

                Packages

                Version Package
                7.0.0 DXP FP9
                7.0.0 DXP SP2
                7.0.3 CE GA4
                7.1.X
                Master