Exceptions thrown in Elasticsearch initialization thread are ignored

• Type: Regression Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.1.X, 7.2.X, Master
• Fix Version/s: 7.1.10 DXP FP13, 7.1.X, 7.2.10 DXP FP1, 7.2.1 CE GA2, 7.2.X, 7.3.10 DXP GA1, Master
• Component/s: Search Infrastructure > Elasticsearch Connector
• Labels:
  • ee-ts
  • liferay-fixpack-dxp-1-7210
  • liferay-fixpack-dxp-13-7110
• Branch Version/s:
  7.2.x, 7.1.x
• Backported to Branch:
  Committed
• Fix Priority: 3
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/75246

Exceptions thrown in Elasticsearch initialization thread are ignored, so in case of any error, Elasticsearch connector is started with client = null and lots of NPE are thrown during startup.

Correct behavior should be: catch all startup exceptions and in case of error, don't start Elasticsearch connector in a inconsistent state

Root cause
Caused by LPS-85003, in that LPS following commit added the new "Elastic search initialization thread":

• https://github.com/liferay/liferay-portal/commit/243dd9dc6bcb85558a58fe3fd261e931d81ecf05

Steps to reproduce

1. Setup a Liferay installation.
2. Start it and verify that [LIFERAY_HOME]/data/elasticsearch6 folder was created and no error is thrown during startup
3. Stop Liferay
4. Go to [LIFERAY_HOME]/data and execute following commands in order to force an Elasticsearch initialization error

   1. mv elasticsearch6 elasticsearch6_old

   2. touch elasticsearch6

5. Start Liferay again and check log file
   • Expected behavior: only one ERROR trace and no WARN traces are present in log startup
   • Wrong behavior: several ERROR and WARN traces are present in log startup

     2019-06-28 14:05:10.242 WARN  [liferay/search_writer/SYSTEM_ENGINE-5][ProxyMessageListener:88] java.lang.NullPointerException
     java.lang.NullPointerException
             at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
             at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.RefreshIndexRequestExecutorImpl.execute(RefreshIndexRequestExecutorImpl.java:46)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.ElasticsearchIndexRequestExecutor.executeIndexRequest(ElasticsearchIndexRequestExecutor.java:140)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:39)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:24)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:67)
             at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.commit(ElasticsearchIndexWriter.java:109)
             at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
             at java.lang.reflect.Method.invoke(Method.java:498)
             at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
             at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
             at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
             at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
             at java.lang.Thread.run(Thread.java:748)
     2019-06-28 14:05:10.242 WARN  [liferay/search_writer/SYSTEM_ENGINE-1][ProxyMessageListener:88] java.lang.NullPointerException
     java.lang.NullPointerException
             at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
             at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.BulkDocumentRequestExecutorImpl.execute(BulkDocumentRequestExecutorImpl.java:56)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.ElasticsearchDocumentRequestExecutor.executeBulkDocumentRequest(ElasticsearchDocumentRequestExecutor.java:48)
             at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:30)
             at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:23)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:62)
             at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.updateDocument(ElasticsearchIndexWriter.java:272)
             at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
             at java.lang.reflect.Method.invoke(Method.java:498)
             at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
             at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
             at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
             at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
             at java.lang.Thread.run(Thread.java:748)
     2019-06-28 14:05:10.246 WARN  [liferay/search_writer/SYSTEM_ENGINE-3][ProxyMessageListener:88] java.lang.NullPointerException
     java.lang.NullPointerException
             at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
             at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.BulkDocumentRequestExecutorImpl.execute(BulkDocumentRequestExecutorImpl.java:56)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.document.ElasticsearchDocumentRequestExecutor.executeBulkDocumentRequest(ElasticsearchDocumentRequestExecutor.java:48)
             at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:30)
             at com.liferay.portal.search.engine.adapter.document.BulkDocumentRequest.accept(BulkDocumentRequest.java:23)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:62)
             at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.updateDocument(ElasticsearchIndexWriter.java:272)
             at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
             at java.lang.reflect.Method.invoke(Method.java:498)
             at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
             at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
             at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
             at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
             at java.lang.Thread.run(Thread.java:748)
     2019-06-28 14:05:10.246 WARN  [liferay/search_writer/SYSTEM_ENGINE-1][ProxyMessageListener:88] java.lang.NullPointerException
     java.lang.NullPointerException
             at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
             at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.RefreshIndexRequestExecutorImpl.execute(RefreshIndexRequestExecutorImpl.java:46)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.ElasticsearchIndexRequestExecutor.executeIndexRequest(ElasticsearchIndexRequestExecutor.java:140)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:39)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:24)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:67)
             at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.commit(ElasticsearchIndexWriter.java:109)
             at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
             at java.lang.reflect.Method.invoke(Method.java:498)
             at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
             at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
             at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
             at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
             at java.lang.Thread.run(Thread.java:748)
     2019-06-28 14:05:10.246 WARN  [liferay/search_writer/SYSTEM_ENGINE-4][ProxyMessageListener:88] java.lang.NullPointerException
     java.lang.NullPointerException
             at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
             at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.RefreshIndexRequestExecutorImpl.execute(RefreshIndexRequestExecutorImpl.java:46)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.index.ElasticsearchIndexRequestExecutor.executeIndexRequest(ElasticsearchIndexRequestExecutor.java:140)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:39)
             at com.liferay.portal.search.engine.adapter.index.RefreshIndexRequest.accept(RefreshIndexRequest.java:24)
             at com.liferay.portal.search.elasticsearch6.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:67)
             at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchIndexWriter.commit(ElasticsearchIndexWriter.java:109)
             at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
             at java.lang.reflect.Method.invoke(Method.java:498)
             at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
             at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
             at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
             at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
             at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
             at java.lang.Thread.run(Thread.java:748)