Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-113128

Liferay index intermittently isn't replicated after reindexing

    Details

      Description

      To recover from this bug, manually create a follower index for the liferay company index if it is missing after reindexing.

       

      Incidences:

      Date Test Name Location Jenkins
       5/6/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search  link
       5/5/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search link
      5/13/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search link
      5/15/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search link
      5/18/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search link
      5/27/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search link
      6/3/2020 Elasticsearch6EE#AssertReplicationAfterReindex  7.2.x ci:test:search  link

      Steps to reproduce:

      1. Set up two Elasticsearch clusters for with their trial licenses activated by running the following command from your portal root directory: "ant -f build-test-elasticsearch6.xml start-elasticsearch -Dremote.elasticsearch.ccr.enabled=true"
        1. This also sets "index.soft_deletes.enabled: true" in "Additional Index Configurations"
      2. Start up portal
      3. Create a Web Content Article titled "Apple 1"
      4. Navigate to Control Panel > Configuration > System Settings > Search > Cross-Cluster Replication Elasticsearch Connections
      5. Add a new connection with the following:
        1. Connection ID: ccr
        2. Cluster Name: LiferayElasticsearchClusterTwo
        3. Transport Address: localhost:9500
        4. Network Host Address: http://localhost:9202
      6. Navigate to Control Panel > Configuration> System Settings > Search > Index Actions
      7. Reindex all search indexes
      8. Connect the Elasticsearch clusters and create follow indices by running the following command from your portal root directory: "ant -f build-test-elasticsearch6.xml create-follower-indices"
      9. Navigate to Control Panel > Configuration > System Settings > Search > Cross-Cluster Replication
      10. Set the following and click "Save":
        1. Check the "Enabled" checkbox
        2. Cross-Cluster Replication Local Cluster Connection Configurations: localhost:8080=ccr
      11. Search for "apple" and assert "Apple 1" is returned as a search result
      12. Navigate to Control Panel > Configuration > Search
      13. Reindex all search indexes
      14. Repeat steps 11 to 13
        1. This step may take a while. Previous manual attempts generally have taken 4 to 15 iterations to reproduce this bug.

      Expected results:
      Expected all indexes to be replicated after reindexing.

      Actual results:
      The liferay company index isn't replicated after reindexing.

      Example stacktrace:

      14:52:53,193 ERROR [http-nio-8080-exec-4][ElasticsearchIndexSearcher:169] java.lang.RuntimeException: org.elasticsearch.index.IndexNotFoundException: [liferay-20101] IndexNotFoundException[no such index]
      java.lang.RuntimeException: org.elasticsearch.index.IndexNotFoundException: [liferay-20101] IndexNotFoundException[no such index]
      	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:752)
      	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:704)
      	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:660)
      	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:168)
      	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:143)
      	at org.elasticsearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:289)
      	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:303)
      	at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:195)
      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
      	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114)
      	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87)
      	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217)
      	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:70)
      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:121)
      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
      	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
      	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:89)
      	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:80)
      	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250)
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308)
      	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
      	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:193)
      	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1046)
      	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:932)
      	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763)
      	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
      	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
      	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
      	at java.lang.Thread.run(Thread.java:745) 

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              timothy.pak Timothy Pak
              Reporter:
              brooke.dalton Brooke Dalton
              Participants of an Issue:
              Recent user:
              Bryan Engler
              Engineering Assignee:
              Bryan Engler
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                1 year, 16 weeks, 5 days ago

                  Packages

                  Version Package
                  7.2.10 DXP FP8
                  7.2.10.3 DXP SP3
                  7.2.X
                  7.3.5 CE GA6
                  7.3.10 DXP GA1
                  Master