Tomcat does not shut down after connecting to remote Elasticsearch 7

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.2.X, Master
• Fix Version/s: 7.2.10 DXP FP7, 7.2.X, 7.3.3 CE GA4, 7.3.10 DXP GA1, Master
• Component/s: Search Infrastructure
• Labels:
  • backend_test
  • ee-ts
  • liferay-fixpack-dxp-7-7210
  • marketplace-portal-search-elasticsearch7-3.1.0-7210
• Branch Version/s:
  7.2.x
• Backported to Branch:
  Committed
• Fix Priority: 3
• Epic Link: Elasticsearch 7 upgrade (Phase 2)
• Sprint: Search|S03 Sprint 14| 4.6-4.27, Search|S03 Sprint 15|4.28-5.18, Search|S03 Sprint 16|5.18-6.8
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/89396

Patch Availability

• Connector to Elasticsearch 6: Bundled -> Fix Pack 7+
• Connector to Elasticsearch 7: Marketplace -> To be released (v3.0.1 is still affected)

Search Engines may cause tomcat to hang during the shutdown.sh process and never fully end its process. This can be revealed by checking if the java process is still running after the shutdown.sh command has been issued to tomcat after a sufficient waiting time. We currently are unable to reproduce this issue and no errors are generated in the logs, but when it does happen thread dumps will look like this:

Start Level: Equinox Container: dc420add-9545-47f2-8d9b-2aa55aff02c8
Java Stack
at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) 
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.5/LockSupport.java:234) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.5/AbstractQueuedSynchronizer.java:2211) 
at com.liferay.portal.kernel.util.ServiceProxyFactory$AwaitServiceInvocationHandler.invoke(ServiceProxyFactory.java:250) 
at com.sun.proxy.$Proxy250.createDestination(Unknown Source) 
at com.liferay.portal.kernel.messaging.DestinationFactoryUtil.createDestination(DestinationFactoryUtil.java:29) 
at com.liferay.portal.kernel.search.BaseSearchEngineConfigurator.createSearchReaderDestination(BaseSearchEngineConfigurator.java:149) 
at com.liferay.portal.kernel.search.BaseSearchEngineConfigurator.getSearchReaderDestination(BaseSearchEngineConfigurator.java:278) 
at com.liferay.portal.kernel.search.BaseSearchEngineConfigurator.destroySearchEngine(BaseSearchEngineConfigurator.java:209) 
at com.liferay.portal.kernel.search.BaseSearchEngineConfigurator.destroy(BaseSearchEngineConfigurator.java:94) 
at com.liferay.portal.search.elasticsearch7.internal.ElasticsearchEngineConfigurator.destroy(ElasticsearchEngineConfigurator.java:53) 
at com.liferay.portal.search.internal.SearchEngineHelperImpl$1.removedService(SearchEngineHelperImpl.java:324) 
at com.liferay.portal.search.internal.SearchEngineHelperImpl$1.removedService(SearchEngineHelperImpl.java:292) 
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:969) 
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1) 
at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341) 
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:911) 
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) 
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:891) 
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) 
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:222) 
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:927) 
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:892) 
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:135) 
at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:969)
.....