Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-85215

"SearchException: Unable to delete document" occurs on Portal server shutdown

    Details

      Description

      Steps to reproduce (inconsistent):

      1. Start up a fresh Portal bundle
      2. Add some assets (e.g. Users, Knowledge Base Article)
      3. Delete the assets
      4. Shut down portal

      Result:
      Stacktrace warnings show up in the console log. Example:

      2018-09-07 12:31:35.790 WARN  [liferay/search_writer/SYSTEM_ENGINE-7][ProxyMessageListener:88] com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.user.associated.data.web.internal.configuration.AnonymousUserConfiguration
      com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.user.associated.data.web.internal.configuration.AnonymousUserConfiguration
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:138)
      	at sun.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: Future got interrupted
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:125)
      	... 10 more
      Caused by: java.lang.InterruptedException
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)
      	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)
      	... 12 more
      2018-09-07 12:31:35.790 WARN  [liferay/search_writer/SYSTEM_ENGINE-4][ProxyMessageListener:88] com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBDisplayPortletInstanceConfiguration
      com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBDisplayPortletInstanceConfiguration
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:138)
      	at sun.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: Future got interrupted
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:125)
      	... 10 more
      Caused by: java.lang.InterruptedException
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)
      	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)
      	... 12 more
      2018-09-07 12:31:35.795 WARN  [liferay/search_writer/SYSTEM_ENGINE-2][ProxyMessageListener:88] com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBSearchPortletInstanceConfiguration
      com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBSearchPortletInstanceConfiguration
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:138)
      	at sun.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: Future got interrupted
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:125)
      	... 10 more
      Caused by: java.lang.InterruptedException
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)
      	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)
      	... 12 more
      2018-09-07 12:31:35.795 WARN  [liferay/search_writer/SYSTEM_ENGINE-6][ProxyMessageListener:88] com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBArticlePortletInstanceConfiguration
      com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBArticlePortletInstanceConfiguration
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:138)
      	at sun.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: Future got interrupted
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:125)
      	... 10 more
      Caused by: java.lang.InterruptedException
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)
      	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)
      	... 12 more
      2018-09-07 12:31:35.794 WARN  [liferay/search_writer/SYSTEM_ENGINE-8][ProxyMessageListener:88] com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBSectionPortletInstanceConfiguration
      com.liferay.portal.kernel.search.SearchException: Unable to delete document com_liferay_configuration_admin_web_portlet_SystemSettingsPortlet_PORTLET_com.liferay.knowledge.base.web.internal.configuration.KBSectionPortletInstanceConfiguration
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:138)
      	at sun.reflect.GeneratedMethodAccessor694.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: Future got interrupted
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:42)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
      	at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.deleteDocument(ElasticsearchIndexWriter.java:125)
      	... 10 more
      Caused by: java.lang.InterruptedException
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)
      	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
      	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)
      	... 12 more
      

      Note:
      This happens inconsistently and has no functional impact. We believe this issue only occurs on release bundles (i.e. 7.1 GA1 + FP)
      Per Andre:
      It looks like something that could happen if...
      1. During shutdown, portlets being undeployed afterwards generate a "delete document" command (because besides regular user data, we index portlet metadata too)
      2. The commands pile up in the command queue buffer
      3. Eventually the embedded Elasticsearch node is closed as part of shutdown
      4. Commands left in the queue at this point will be consumed and crash against closed Elasticsearch

      Reproduced on:
      WebLogic Server 12c + IBM DB2 11.1 + Open JDK8 + CentOS 7 64-Bit
      Portal GA1 7.1.10 + Fix Pack 2 (Build2)

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  24 weeks, 2 days ago

                  Packages

                  Version Package