Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-119745

[CCR] Warning seen on the follower cluster console after enabling Cross-Cluster Replication

    Details

    • Type: Bug
    • Status: Verified
    • Resolution: Unresolved
    • Affects Version/s: 7.2.X
    • Fix Version/s: None
    • Component/s: Search Infrastructure
    • Labels:
      None

      Description

      Workaround: Add ilm-history-2-000001 to Excluded Indexes on the Cross Cluster Replication page before enabling CCR in step 7.

      Steps to reproduce:

      1. Setup two Elasticsearch clusters for CCR
        1. ant -f build-test-elasticsearch7.xml start-elasticsearch -Dremote.elasticsearch.ccr.enabled=true
      2. Deploy Elasticsearch 7 to the bundle and blacklist Elasticsearch 6 modules
        1. ant -f build-test-elasticsearch7.xml deploy-elasticsearch7
      3. Start portal
      4. Navigate to Control Panel > Configuration > System Settings > Search > Cross-Cluster Replication Elasticsearch Connections
      5. Add a new connection with the following:
        1. Connection ID: ccr
        2. Cluster Name: LiferayElasticsearchClusterTwo
        3. Transport Address: localhost:9500
        4. Network Host Address: http://localhost:9202
      6. Navigate to Control Panel > Configuration > System Settings > Search > Cross-Cluster Replication
      7. Set the following and click "Save":
        1. Check the "Enabled" checkbox
        2. Cross-Cluster Replication Local Cluster Connection Configurations: localhost:8080,ccr

      Expected result:
      Expected no warnings to be seen in the Elasticsearch console

      Actual result:
      Following is thrown in the follower cluster console. It appears that the ilm-history-2-000001 index is being replicated. No functional impact.

      [2020-08-24T13:55:45,311][WARN ][o.e.x.i.h.ILMHistoryStore] [tim] failed to create ILM history store index prior to issuing bulk request
      java.lang.IllegalStateException: ILM history alias [ilm-history-2does not have a write index
      	at org.elasticsearch.xpack.ilm.history.ILMHistoryStore.ensureHistoryIndex(ILMHistoryStore.java:219) [x-pack-ilm-7.7.0.jar:7.7.0]
      	at org.elasticsearch.xpack.ilm.history.ILMHistoryStore$1.beforeBulk(ILMHistoryStore.java:82) [x-pack-ilm-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:58) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:455) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:464) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.access$400(BulkProcessor.java:54) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor$Flush.run(BulkProcessor.java:504) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:223) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_265]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_265]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
      	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
      [2020-08-24T13:55:45,331][WARN ][o.e.x.i.h.ILMHistoryStore] [tim] unable to index the following ILM history items:
      {index_age=208268, @timestamp=1598302542812, success=true, index=ilm-history-2-000001, state={phase=new, phase_definition={"policy":"ilm-history-ilm-policy","version":1,"modified_date_in_millis":1598302231241}, action_time=1598302542752, phase_time=1598302542752, action=complete, step=complete, creation_date=1598302334544, step_time=1598302542752}, policy=ilm-history-ilm-policy}
      {index_age=208317, @timestamp=1598302542861, success=true, index=ilm-history-2-000001, state={phase=hot, phase_definition={"policy":"ilm-history-ilm-policy","phase_definition":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"version":1,"modified_date_in_millis":1598302231241}, action_time=1598302542817, phase_time=1598302542817, action=unfollow, step=wait-for-indexing-complete, creation_date=1598302334544, step_time=1598302542817}, policy=ilm-history-ilm-policy}
      {index_age=208388, @timestamp=1598302542932, success=true, index=ilm-history-2-000001, state={phase=hot, phase_definition={"policy":"ilm-history-ilm-policy","phase_definition":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"version":1,"modified_date_in_millis":1598302231241}, action_time=1598302542817, phase_time=1598302542817, step_info={"message":"waiting for the [index.lifecycle.indexing_complete] setting to be set to true on the leader index, it is currently [false]","index.lifecycle.indexing_complete":false}, action=unfollow, step=wait-for-indexing-complete, creation_date=1598302334544, step_time=1598302542817}, policy=ilm-history-ilm-policy}
      java.util.concurrent.ExecutionException: java.lang.IllegalStateException: ILM history alias [ilm-history-2does not have a write index
      	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_265]
      	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) ~[?:1.8.0_265]
      	at org.elasticsearch.xpack.ilm.history.ILMHistoryStore$1.beforeBulk(ILMHistoryStore.java:87) [x-pack-ilm-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:58) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:455) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:464) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor.access$400(BulkProcessor.java:54) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.action.bulk.BulkProcessor$Flush.run(BulkProcessor.java:504) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:223) [elasticsearch-7.7.0.jar:7.7.0]
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_265]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_265]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_265]
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_265]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
      	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
      Caused by: java.lang.IllegalStateException: ILM history alias [ilm-history-2does not have a write index
      	at org.elasticsearch.xpack.ilm.history.ILMHistoryStore.ensureHistoryIndex(ILMHistoryStore.java:219) ~[?:?]
      	at org.elasticsearch.xpack.ilm.history.ILMHistoryStore$1.beforeBulk(ILMHistoryStore.java:82) ~[?:?]
      	... 14 more 
      

      Reproduced on:
      Tomcat 9.0.33 + MySQL 5.7
      Portal 7.2.x GIT ID: cbffa6f144dda1dc5b471c7bb829b8f61850f2f7

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              support-lep@liferay.com SE Support
              Reporter:
              timothy.pak Timothy Pak
              Participants of an Issue:
              Recent user:
              Joshua Chong
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Dates

                Created:
                Updated:
                Days since last comment:
                47 weeks, 4 days ago

                  Packages

                  Version Package