Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-120240

Workflow Metrics loads indefinitely on Elasticsearch 7.8+

    Details

      Description

      Solution Notes & Workaround

      Please note that as part of the changes, the client version used by the Elasticsearch 7 connector was also updated and it uses 7.9.0 as of "Liferay Connector to Elasticsearch 7 v3.2.0. The compatible Elasticsearch version shifts from "7.3.x through 7.8.x" to "7.9.x".
      If you want to understand which factors influence Liferay DXP's compatibility with Elasticsearch, please read this article in Help Center.

      As a workaround, you can use an ES7 server with version below 7.8.0. Will be fixed in Elasticsearch 7.10.0.


      Liferay Connector to Elasticsearch 7:
      Affects: 3.1.0 and below.
      Fixed in: 3.2.0+


      Steps to reproduce:

      1. Deploy Elasticsearch 7 to the bundle
        1. ant -f build-test-elasticsearch7.xml deploy-elasticsearch7
      2. Setup and start a remote Elasticsearch 7 server that's 7.8.0 or newer (7.2.x build scripts sets up Elasticsearch 7.9.0)
        1. ant -f build-test-elasticsearch7.xml start-elasticsearch
      3. Start portal
      4. Navigate to Control Panel > Workflow > Metrics

      Expected results:
      Expected the Metrics page to load after a reasonable amount of time

      Actual result:
      Metrics page loads indefinitely.

      The following stacktrace is thrown in the Liferay logs:

      2020-09-01 20:46:28.241 ERROR [http-nio-8080-exec-4][ExceptionMapper:31] NoNodeAvailableException[None of the configured nodes were available: [{tim}{QvaSyDPWRymFRBcKzQKjyA}{oLCis41aRkm24QEjJiojAA}{127.0.0.1}{127.0.0.1:9300}{dimrt}{xpack.installed=true, transform.node=true}]]; nested: NodeDisconnectedException[[tim][127.0.0.1:9300][indices:data/read/search] disconnected];
      NoNodeAvailableException[None of the configured nodes were available: [{tim}{QvaSyDPWRymFRBcKzQKjyA}{oLCis41aRkm24QEjJiojAA}{127.0.0.1}{127.0.0.1:9300}{dimrt}{xpack.installed=true, transform.node=true}]]; nested: NodeDisconnectedException[[tim][127.0.0.1:9300][indices:data/read/search] disconnected];
      	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:294)
      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)
      	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111)
      	at org.elasticsearch.transport.TransportService$8.run(TransportService.java:973)
      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: NodeDisconnectedException[[tim][127.0.0.1:9300][indices:data/read/search] disconnected]
      

      The following stacktrace is thrown in the Elasticsearch logs

      [2020-09-01T13:46:28,219][WARN ][o.e.t.OutboundHandler    ] [tim] send message failed [channel: Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:45856}]
      java.lang.IllegalArgumentException: scripted_metric doesn't support cross cluster search until 7.8.0
      	at org.elasticsearch.search.aggregations.metrics.InternalScriptedMetric.doWriteTo(InternalScriptedMetric.java:79) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.doWriteTo(InternalSingleBucketAggregation.java:73) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.writeTo(InternalTerms.java:96) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeCollection$31(StreamOutput.java:1160) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1179) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1160) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeList(StreamOutput.java:1167) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalMappedTerms.writeTermTypeInfoTo(InternalMappedTerms.java:80) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doWriteTo(InternalTerms.java:181) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:952) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:67) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:374) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:159) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:144) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:197) [elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$WriteOperation.buffer(Netty4MessageChannelHandler.java:213) [transport-netty4-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.doFlush(Netty4MessageChannelHandler.java:147) [transport-netty4-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.flush(Netty4MessageChannelHandler.java:117) [transport-netty4-client-7.9.0.jar:7.9.0]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
      [2020-09-01T13:46:28,225][WARN ][o.e.t.TcpTransport       ] [tim] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:45856}], closing connection
      java.lang.IllegalArgumentException: scripted_metric doesn't support cross cluster search until 7.8.0
      	at org.elasticsearch.search.aggregations.metrics.InternalScriptedMetric.doWriteTo(InternalScriptedMetric.java:79) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.doWriteTo(InternalSingleBucketAggregation.java:73) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.writeTo(InternalTerms.java:96) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeCollection$31(StreamOutput.java:1160) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1179) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeCollection(StreamOutput.java:1160) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeList(StreamOutput.java:1167) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalMappedTerms.writeTermTypeInfoTo(InternalMappedTerms.java:80) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doWriteTo(InternalTerms.java:181) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregation.writeTo(InternalAggregation.java:202) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1091) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1216) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:115) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:952) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:67) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:374) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:159) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:144) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:197) ~[elasticsearch-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$WriteOperation.buffer(Netty4MessageChannelHandler.java:213) ~[transport-netty4-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.doFlush(Netty4MessageChannelHandler.java:147) ~[transport-netty4-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.flush(Netty4MessageChannelHandler.java:117) ~[transport-netty4-client-7.9.0.jar:7.9.0]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
      	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
      

      Reproduced on:
      Tomcat 9.0.33 + MySQL 5.7
      Portal 7.2.x GIT ID: 2066dc3adbca06dd46aaaf3a608582fb1799f676
      Elasticsearch 7.8.0
      Elasticsearch 7.9.0

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              timothy.pak Timothy Pak
              Reporter:
              timothy.pak Timothy Pak
              Participants of an Issue:
              Recent user:
              Tibor Lipusz
              Engineering Assignee:
              Bryan Engler
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                24 weeks, 4 days ago
                Development End Date:
                Development Start Date:

                  Packages

                  Version Package
                  7.1.10 DXP FP20
                  7.1.10.5 SP5
                  7.1.X
                  7.2.X