ConcurrentModificationException during portal-workflow-kaleo-forms-web upgrade process

• Type: Regression Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.X EE
• Fix Version/s: 7.0.0 DXP FP25, 7.0.X EE, 7.0.4 CE GA5
• Component/s: Business Productivity > Kaleo Workflow Forms, Workflow > Upgrade
• Labels:
  • QA-AUTOMATION
  • fw_code_review
  • fw_test_unit
  • liferay-digital-enterprise-70-sp5
  • liferay-fixpack-de-25-7010
• Branch Version/s:
  7.0.x
• Backported to Branch:
  Committed
• Story Points: 1.25
• Fix Priority: 3
• Git Pull Request: https://github.com/liferay/liferay-portal-ee/pull/1321

Description
During the v1_0_2/UpgradePortletId#getNewTypeSettings method inside the portal-workflow-kaleo-forms-web module,

1. Start up Liferay 6.2 and log in as the admin user.
2. Deploy kaleo-designer-portlet, kaleo-forms-portlet, and kaleo-web.
3. Add a new page with a 2-column layout.
4. Navigate to the new page and add a single Kaleo Forms portlet to the second column. DO NOT ADD ANY OTHER PORTLETS TO THIS COLUMN (IMPORTANT!)
5. Add a different portlet (it shouldn't matter which one) to the first column.
6. Shut down Liferay.
7. Upgrade your database to Liferay 7 and observe the logs during the upgrade process.

Expected Result: The upgrade process completes successfully with no errors.
Actual Result: The following error occurs during the upgrade process:

java.lang.RuntimeException: com.liferay.portal.kernel.upgrade.UpgradeException: java.util.ConcurrentModificationException
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands$UpgradeInfosRunnable.run(ReleaseManagerOSGiCommands.java:490)
        at com.liferay.portal.output.stream.container.OutputStreamContainerFactoryTracker.runWithSwappedLog(OutputStreamContainerFactoryTracker.java:100)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands.executeUpgradeInfos(ReleaseManagerOSGiCommands.java:294)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands.doExecute(ReleaseManagerOSGiCommands.java:250)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands$UpgradeInfoServiceTrackerMapListener.keyEmitted(ReleaseManagerOSGiCommands.java:440)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands$UpgradeInfoServiceTrackerMapListener.keyEmitted(ReleaseManagerOSGiCommands.java:430)
        at com.liferay.osgi.service.tracker.collections.internal.map.ServiceTrackerMapImpl$DefaultEmitter.emit(ServiceTrackerMapImpl.java:222)
        at com.liferay.osgi.service.tracker.collections.map.PropertyServiceReferenceMapper.map(PropertyServiceReferenceMapper.java:43)
        at com.liferay.osgi.service.tracker.collections.internal.map.ServiceTrackerMapImpl$ServiceReferenceServiceTrackerCustomizer.addingService(ServiceTrackerMapImpl.java:261)
        at com.liferay.osgi.service.tracker.collections.internal.map.ServiceTrackerMapImpl$ServiceReferenceServiceTrackerCustomizer.addingService(ServiceTrackerMapImpl.java:248)
        at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)
        at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:1)
        at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
        at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)
        at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:318)
        at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)
        at com.liferay.osgi.service.tracker.collections.internal.map.ServiceTrackerMapImpl.open(ServiceTrackerMapImpl.java:94)
        at com.liferay.osgi.service.tracker.collections.map.ServiceTrackerMapFactory.openMultiValueMap(ServiceTrackerMapFactory.java:244)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands.activate(ReleaseManagerOSGiCommands.java:213)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:224)
        at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39)
        at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:617)
        at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:501)
        at org.apache.felix.scr.impl.inject.ActivateMethod.invoke(ActivateMethod.java:302)
        at org.apache.felix.scr.impl.inject.ActivateMethod.invoke(ActivateMethod.java:294)
        at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:297)
        at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:108)
        at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:906)
        at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:879)
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:748)
        at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
        at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
        at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:886)
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:873)
        at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:132)
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:940)
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:740)
        at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1012)
        at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:968)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136)
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945)
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881)
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167)
        at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:464)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:482)
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:1001)
        at com.liferay.registry.internal.RegistryImpl.registerService(RegistryImpl.java:305)
        at com.liferay.portal.tools.DBUpgrader._registerModuleServiceLifecycle(DBUpgrader.java:410)
        at com.liferay.portal.tools.DBUpgrader.main(DBUpgrader.java:106)
Caused by: com.liferay.portal.kernel.upgrade.UpgradeException: java.util.ConcurrentModificationException
        at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:92)
        at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:123)
        at com.liferay.portal.upgrade.internal.release.ReleaseManagerOSGiCommands$UpgradeInfosRunnable.run(ReleaseManagerOSGiCommands.java:469)
        ... 77 more
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
        at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
        at com.liferay.portal.workflow.kaleo.forms.web.internal.upgrade.v1_0_2.UpgradePortletId.getNewTypeSettings(UpgradePortletId.java:62)
        at com.liferay.portal.workflow.kaleo.forms.web.internal.upgrade.v1_0_2.UpgradePortletId.removePortletIdFromLayouts(UpgradePortletId.java:115)
        at com.liferay.portal.workflow.kaleo.forms.web.internal.upgrade.v1_0_2.UpgradePortletId.deletePortletReferences(UpgradePortletId.java:45)
        at com.liferay.portal.workflow.kaleo.forms.web.internal.upgrade.v1_0_2.UpgradePortletId.doUpgrade(UpgradePortletId.java:52)
        at com.liferay.portal.kernel.upgrade.UpgradeProcess.upgrade(UpgradeProcess.java:89)
        ... 79 more

If you run upgrade:check from gogo shell, you can see that the upgrade process for com.liferay.portal.workflow.kaleo.forms.web did not complete successfully.

Note: The issue may be reproduced inconsistently because the HashMap iterator needs to not hit the column you added the Kaleo Forms portlet to last. With the setup I described in the steps above (two column layout with column-1 not empty and column-2 containing a Kaleo Forms portlet), I was able to reproduce the issue. I believe it should be consistent because each of those keys should hash to the same value and so I would expect the HashMap iterator to iterate over them in a consistent order, but I am not 100% sure on this. So if you don't reproduce the issue, you can just retry a few times.

Not applicable to master because the Kaleo Forms portlet does not exist in master. However, the issue does exist in the master-private repo, so we will fix the issue there (a30466b4f1839d2df87e8c20c4172229cf95b9d3)
Reproduced in ee-7.0.x (8a8aVisual64b34b86c17264b39d950e4e3e7c790eb2f8)