Intermittent Elasticsearch 7 test failures due to MasterNotDiscoveredException

• Type: Regression Bug
• Status: Verified
• Resolution: Unresolved
• Affects Version/s: 7.2.X, Master
• Fix Version/s: None
• Component/s: Search Infrastructure
• Labels:
  • 7.3-known-issues
  • 7.3-won't
  • QA-FP
  • backend_test
  • flaky_test_search
  • liferay-ga6-ce-735
  • liferay-ga7-ce-736
  • liferay-ga8-ce-737
• Fix Priority: 4
• Epic Link: Flow Forward
• Sprint: Search|S03 Sprint 18|6.29-7.20, Search|S03 Sprint 19|7.21-8.10, Search|S03 Sprint 20 | 8.10-31
• QA Test Name:
  • lpkg-persistence-jdk8/0
• QA Test Score: 5

master

lpkg-controller-jdk8 incidences:

Date	Test Name	Location	Full Console
6/24/2020	lpkg-controller-jdk8 PortalLogAssertorTest	ci:forward	console txt

Unit test incidences:

Date	Test Name	Location	Full Console
8/5/2020	Cluster2InstancesTest.testExpandAndShrink ElasticsearchHighlightTest.testEllipsisElasticsearch HighlighterTest.testHighlight StringQueryTest.testBooleanOperatorNotDeepElasticsearch	ci:test:search	console text

7.2.x
Unit test incidences:

Date	Test Name	Location	Full Console
3/10/2020	MultiValueFacetTest.testSelection	7.2.x PR	link
3/11/2020	MaxBucketPipelineAggregationTest.testMaxBucketPipeline	7.2.x PR	link
6/16/2020	WorkflowTaskManagerImplTest.testApproveDLFileEntryInDLFolderWhenHomeDLFolderHasWorkflow	ci:test:search-es7	link

Example stacktrace:

MasterNotDiscoveredException[null]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251)
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325)
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252)
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:572)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Functional test incidences:

Date	Test Name	Location	Full Console
4/2/2020	Search#SearchDocumentsAndMediaDocument	ci:test:search-es7	04022020

Example Stacktrace:

13:45:39,026 ERROR [main][PortalInstances:271] MasterNotDiscoveredException[FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];]; nested: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
MasterNotDiscoveredException[FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];]; nested: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251)
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325)
	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:149)
	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:120)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retry(TransportMasterNodeAction.java:235)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:192)
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:93)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70)
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70)
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:79)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.onFailure(MetaDataCreateIndexService.java:623)
	at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.onFailure(MasterService.java:499)
	at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$publishingFailed$0(MasterService.java:403)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at org.elasticsearch.cluster.service.MasterService$TaskOutputs.publishingFailed(MasterService.java:403)
	at org.elasticsearch.cluster.service.MasterService.onPublicationFailed(MasterService.java:287)
	at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:259)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1378)
	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)
	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54)
	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1302)
	at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124)
	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88)
	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
	... 3 more
Caused by: ElasticsearchException[publication cancelled before committing: timed out after 30s]
	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85)
	... 5 more