Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-124213

Elasticsearch 7.10.0 breaks Workflow Metrics

    Details

      Description

      Works with 7.9.0 through 7.9.3. DXP 7.2 does not seem to be affected either when connecting to ES 7.10.0.
      Related Elasticsearch issue (they won't restore the old behavior though): https://github.com/elastic/elasticsearch/issues/66057


      Steps to reproduce:

      1. Start up a remote Elasticsearch 7.10.0 server and set it up Portal to connect to it
        1. Before starting Portal, change this Elasticsearch version within test.properties to 7.10.0
        2. Execute ant -f build-test-elasticsearch7.xml start-elasticsearch
      2. Start up Portal
      3. Reindex search indexes
      4. Navigate to Applications Menu > Workflow > Metrics
      5. Reindex Workflow Metrics indexes through the settings menu
      6. Click on "Single Approver"

      Result: The errors below appear in the console. The functional impact consists of various elements within Workflow Metrics failing to load depending on what the user is doing.

      2020-12-01 01:18:42.824 ERROR [http-nio-8080-exec-4][ExceptionMapper:31] ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]
      ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]]
      	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187)
      	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
      	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
      	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
      	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
      	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
      	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl.getSearchResponse(SearchSearchRequestExecutorImpl.java:95)
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl.execute(SearchSearchRequestExecutorImpl.java:66)
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.ElasticsearchSearchRequestExecutor.executeSearchRequest(ElasticsearchSearchRequestExecutor.java:59)
      	at com.liferay.portal.workflow.metrics.rest.internal.resource.v1_0.NodeMetricResourceImpl._getNodeMetrics(NodeMetricResourceImpl.java:413)
      	at com.liferay.portal.workflow.metrics.rest.internal.resource.v1_0.NodeMetricResourceImpl.getProcessNodeMetricsPage(NodeMetricResourceImpl.java:112)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
      	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
      	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
      	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
      	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
      	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
      	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
      	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
      	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
      	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
      	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
      	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
      	at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216)
      	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
      	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:225)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
      	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
      	at org.eclipse.equinox.http.servlet.internal.registration.EndpointRegistration.service(EndpointRegistration.java:153)
      	at org.eclipse.equinox.http.servlet.internal.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:50)
      	at com.liferay.portal.security.auth.verifier.internal.tracker.AuthVerifierFilterTracker$RemoteAccessFilter.doFilter(AuthVerifierFilterTracker.java:165)
      	at org.eclipse.equinox.http.servlet.internal.registration.FilterRegistration.doFilter(FilterRegistration.java:121)
      	at org.eclipse.equinox.http.servlet.internal.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:45)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:147)
      	at com.liferay.portal.servlet.filters.authverifier.AuthVerifierFilter.processFilter(AuthVerifierFilter.java:196)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at org.eclipse.equinox.http.servlet.internal.registration.FilterRegistration.doFilter(FilterRegistration.java:121)
      	at org.eclipse.equinox.http.servlet.internal.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:45)
      	at org.eclipse.equinox.http.servlet.internal.servlet.ResponseStateHandler.processRequest(ResponseStateHandler.java:70)
      	at org.eclipse.equinox.http.servlet.internal.context.DispatchTargets.doDispatch(DispatchTargets.java:120)
      	at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl.doDispatch(HttpServiceRuntimeImpl.java:373)
      	at org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet.service(ProxyServlet.java:70)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
      	at com.liferay.portal.module.framework.ModuleFrameworkServletAdapter.service(ModuleFrameworkServletAdapter.java:52)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:124)
      	at com.liferay.portal.servlet.filters.password.modified.PasswordModifiedFilter.processFilter(PasswordModifiedFilter.java:62)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:215)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.servlet.filters.lockout.LockoutFilter.processFilter(LockoutFilter.java:58)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:215)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:147)
      	at com.liferay.portal.sharepoint.SharepointFilter.processFilter(SharepointFilter.java:88)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:215)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:147)
      	at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:248)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:215)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:196)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:99)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:196)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:99)
      	at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
      	at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
      	at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
      	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
      	at com.liferay.portal.servlet.filters.urlrewrite.UrlRewriteFilter.processFilter(UrlRewriteFilter.java:65)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:49)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:215)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:175)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:99)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:175)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:99)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:196)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:99)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:104)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
      	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
      	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
      	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
      	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/liferay-20097-workflow-metrics-tasks,liferay-20097-workflow-metrics-sla-task-results/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 500 Internal Server Error]
      {"error":{"root_cause":[{"type":"response_handler_failure_transport_exception","reason":"java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"liferay-20097-workflow-metrics-sla-task-results","node":"tec7Kbs-R8uJs_cjoz57kQ","reason":{"type":"response_handler_failure_transport_exception","reason":"java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.String","caused_by":{"type":"class_cast_exception","reason":"java.lang.Long cannot be cast to java.lang.String"}}}]},"status":500}
      		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
      		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
      		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
      		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
      		... 107 more
      

       

      Example loading error screenshots:

      Reproduced on:
      Portal master git commit 0b62deed530a06aa26b1bd9ec5c278b97b73e3b9
      Portal 7.3.x git commit 705951875f4da0228c2f7b19473e08ca1c3cf24f

      Not reproduced on:
      Portal 7.2.x git commit 6c66fbc86e843da97dfd31cfe44c5e81a1c92dca

        Attachments

        1. bulkedit.mp4
          337 kB
          Joshua Chong
        2. screen1.jpg
          29 kB
          Joshua Chong
        3. screen2.jpg
          45 kB
          Joshua Chong
        4. screen3.jpg
          48 kB
          Joshua Chong

          Issue Links

            Activity

              People

              Assignee:
              joshua.chong Joshua Chong
              Reporter:
              joshua.chong Joshua Chong
              Participants of an Issue:
              Recent user:
              Tibor Lipusz
              Engineering Assignee:
              Rafael Praxedes
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                34 weeks, 5 days ago
                Development End Date:
                Development Start Date:

                  Packages

                  Version Package
                  7.3.10 DXP FP1
                  7.3.10.1 DXP SP1
                  7.3.X
                  Master