Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-110296

Intermittent Elasticsearch 7 test failures due to MasterNotDiscoveredException

    Details

    • Fix Priority:
      4
    • Epic Link:
    • Sprint:
      Search|S03 Sprint 18|6.29-7.20, Search|S03 Sprint 19|7.21-8.10, Search|S03 Sprint 20 | 8.10-31

      Description

      master

      lpkg-controller-jdk8 incidences:

      Date Test Name Location Full Console
      6/24/2020 lpkg-controller-jdk8 PortalLogAssertorTest ci:forward console txt

      Unit test incidences:

      Date Test Name Location Full Console
      8/5/2020 Cluster2InstancesTest.testExpandAndShrink
      ElasticsearchHighlightTest.testEllipsisElasticsearch
      HighlighterTest.testHighlight
      StringQueryTest.testBooleanOperatorNotDeepElasticsearch
      ci:test:search console text

      7.2.x
      Unit test incidences:

      Date Test Name Location Full Console
      3/10/2020  MultiValueFacetTest.testSelection 7.2.x PR  link 
      3/11/2020 MaxBucketPipelineAggregationTest.testMaxBucketPipeline   7.2.x PR  link
      6/16/2020 WorkflowTaskManagerImplTest.testApproveDLFileEntryInDLFolderWhenHomeDLFolderHasWorkflow ci:test:search-es7 link

      Example stacktrace:

      MasterNotDiscoveredException[null]
      	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251)
      	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325)
      	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252)
      	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:572)
      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Functional test incidences:

      Date Test Name Location Full Console
      4/2/2020 Search#SearchDocumentsAndMediaDocument ci:test:search-es7 04022020

      Example Stacktrace:

      13:45:39,026 ERROR [main][PortalInstances:271] MasterNotDiscoveredException[FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];]; nested: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
      MasterNotDiscoveredException[FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];]; nested: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
      	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.onTimeout(TransportMasterNodeAction.java:251)
      	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325)
      	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:149)
      	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:120)
      	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retry(TransportMasterNodeAction.java:235)
      	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:192)
      	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:93)
      	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70)
      	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70)
      	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:79)
      	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.onFailure(MetaDataCreateIndexService.java:623)
      	at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.onFailure(MasterService.java:499)
      	at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$publishingFailed$0(MasterService.java:403)
      	at java.util.ArrayList.forEach(ArrayList.java:1249)
      	at org.elasticsearch.cluster.service.MasterService$TaskOutputs.publishingFailed(MasterService.java:403)
      	at org.elasticsearch.cluster.service.MasterService.onPublicationFailed(MasterService.java:287)
      	at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:259)
      	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:238)
      	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)
      	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
      	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
      	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: FailedToCommitClusterStateException[publication failed]; nested: ElasticsearchException[publication cancelled before committing: timed out after 30s];
      	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$3.onFailure(Coordinator.java:1378)
      	at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:101)
      	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193)
      	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92)
      	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:54)
      	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1302)
      	at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:124)
      	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:88)
      	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$2.run(Coordinator.java:1269)
      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
      	... 3 more
      Caused by: ElasticsearchException[publication cancelled before committing: timed out after 30s]
      	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:85)
      	... 5 more

        Attachments

          Activity

            People

            Assignee:
            andre.oliveira André Ricardo Barreto de Oliveira
            Reporter:
            joshua.chong Joshua Chong
            Participants of an Issue:
            Recent user:
            Sophia Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Days since last comment:
              1 year, 28 weeks, 3 days ago

                Packages

                Version Package