Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-135085

Index error when switching from Sidecar to Production Mode

    Details

      Description

      Description
      Error when switching Elastic Search from Sidecar to Remote with a Liferay portal activated

      Steps to Reproduce

      1. Start Portal with Sidecar
      2. Start a Remote Elastic Search
      3. Switch from Sidecar to Elastic Search "Production Mode Enabled"

      Expected Result
      No warnings/errors 

      Actual Result
      Search functionality is broken since Portal is unable to connect to the Remote Elasticsearch server. The following warnings and errors are thrown in the console log:

      WARN  [http-nio-8080-exec-4][ProxyMessageListener:86] java.lang.RuntimeException: org.elasticsearch.ElasticsearchStatusException: [liferay-0] ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [liferay-0]]]
      java.lang.RuntimeException: org.elasticsearch.ElasticsearchStatusException: [liferay-0] ElasticsearchStatusException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [liferay-0]]]
      

       

      2021-07-02 14:14:58.041 WARN  [CM Event Dispatcher (Fire ConfigurationEvent: pid=com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration)][CompanyIndexFactory:215] Unable to reinitialize index for company 0
      java.lang.RuntimeException: java.net.ConnectException: Connection refused
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.hasIndex(CompanyIndexFactory.java:309) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.createIndices(CompanyIndexFactory.java:92) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.createCompanyIndexes(CompanyIndexFactory.java:211) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.onElasticsearchConfigurationUpdate(CompanyIndexFactory.java:130) ~[bundleFile:?]
      	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_121]
      	at com.liferay.portal.search.elasticsearch7.internal.configuration.ElasticsearchConfigurationWrapper.activate(ElasticsearchConfigurationWrapper.java:279) [bundleFile:?]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:228) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:664) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:510) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:310) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:300) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeModifiedMethod(SingleComponentManager.java:837) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.modify(SingleComponentManager.java:792) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:710) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:674) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.configurationUpdated(ConfigurableComponentHolder.java:435) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport.configurationEvent(RegionConfigurationSupport.java:325) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport$2.configurationEvent(RegionConfigurationSupport.java:119) [bundleFile:?]
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.sendEvent(ConfigurationManager.java:1685) [bundleFile:?]
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.run(ConfigurationManager.java:1637) [bundleFile:?]
      	at org.apache.felix.cm.impl.UpdateThread.run0(UpdateThread.java:138) [bundleFile:?]
      	at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:105) [bundleFile:?]
      	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
      Caused by: java.net.ConnectException: Connection refused
      	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.hasIndex(CompanyIndexFactory.java:305) ~[bundleFile:?]
      	... 27 more
      Caused by: java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_121]
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_121]
      	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
      	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
      	... 1 more
      2021-07-02 14:14:58.092 WARN  [CM Event Dispatcher (Fire ConfigurationEvent: pid=com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration)][CompanyIndexFactory:215] Unable to reinitialize index for company 20100
      java.lang.RuntimeException: java.net.ConnectException: Connection refused
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.hasIndex(CompanyIndexFactory.java:309) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.createIndices(CompanyIndexFactory.java:92) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.createCompanyIndexes(CompanyIndexFactory.java:211) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.onElasticsearchConfigurationUpdate(CompanyIndexFactory.java:130) ~[bundleFile:?]
      	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_121]
      	at com.liferay.portal.search.elasticsearch7.internal.configuration.ElasticsearchConfigurationWrapper.activate(ElasticsearchConfigurationWrapper.java:279) [bundleFile:?]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:228) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:664) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:510) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:310) [bundleFile:?]
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:300) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeModifiedMethod(SingleComponentManager.java:837) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.modify(SingleComponentManager.java:792) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:710) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:674) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.configurationUpdated(ConfigurableComponentHolder.java:435) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport.configurationEvent(RegionConfigurationSupport.java:325) [bundleFile:?]
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport$2.configurationEvent(RegionConfigurationSupport.java:119) [bundleFile:?]
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.sendEvent(ConfigurationManager.java:1685) [bundleFile:?]
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.run(ConfigurationManager.java:1637) [bundleFile:?]
      	at org.apache.felix.cm.impl.UpdateThread.run0(UpdateThread.java:138) [bundleFile:?]
      	at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:105) [bundleFile:?]
      	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
      Caused by: java.net.ConnectException: Connection refused
      	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:849) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[elasticsearch-rest-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974) ~[elasticsearch-rest-high-level-client-7.9.0.jar:7.9.0]
      	at com.liferay.portal.search.elasticsearch7.internal.index.CompanyIndexFactory.hasIndex(CompanyIndexFactory.java:305) ~[bundleFile:?]
      	... 27 more
      Caused by: java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_121]
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_121]
      	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar:4.4.12]
      	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
      	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
      	... 1 more

       

      Stacktrace when performing a search: 

      2021-07-19 15:29:39.191 ERROR [http-nio-8080-exec-2][ElasticsearchIndexSearcher:165] com.liferay.portal.search.elasticsearch7.internal.connection.ElasticsearchConnectionNotInitializedException: Elasticsearch connection not found. Production Mode Enabled: true, Connection ID: null, Prefer Local Cluster: true, Cross-Cluster Replication Enabled: false. Enable INFO logs on class com.liferay.portal.search.elasticsearch7.internal.connection.ElasticsearchConnectionManager for more information
      com.liferay.portal.search.elasticsearch7.internal.connection.ElasticsearchConnectionNotInitializedException: Elasticsearch connection not found. Production Mode Enabled: true, Connection ID: null, Prefer Local Cluster: true, Cross-Cluster Replication Enabled: false. Enable INFO logs on class com.liferay.portal.search.elasticsearch7.internal.connection.ElasticsearchConnectionManager for more information
      	at com.liferay.portal.search.elasticsearch7.internal.connection.ElasticsearchConnectionManager.getRestHighLevelClient(ElasticsearchConnectionManager.java:192) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl.getSearchResponse(SearchSearchRequestExecutorImpl.java:90) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.SearchSearchRequestExecutorImpl.execute(SearchSearchRequestExecutorImpl.java:66) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.search.ElasticsearchSearchRequestExecutor.executeSearchRequest(ElasticsearchSearchRequestExecutor.java:59) ~[bundleFile:?]
      	at com.liferay.portal.search.engine.adapter.search.SearchSearchRequest.accept(SearchSearchRequest.java:45) ~[bundleFile:?]
      	at com.liferay.portal.search.engine.adapter.search.SearchSearchRequest.accept(SearchSearchRequest.java:34) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.search.engine.adapter.ElasticsearchSearchEngineAdapterImpl.execute(ElasticsearchSearchEngineAdapterImpl.java:103) ~[bundleFile:?]
      	at com.liferay.portal.search.elasticsearch7.internal.ElasticsearchIndexSearcher.search(ElasticsearchIndexSearcher.java:127) ~[bundleFile:?]
      
      

      Workaround from this comment:

      1. Go to System Settings > Search > Elasticsearch Connections
      2. Add a new Elasticsearch Connection entry
        1. Within the new entry, check the "Active" checkbox and enter "test" (or any name of choice) for the Connection ID.
        2. Confirm that other settings like "Network Host Addresses" are correct
        3. Save the config
      3. Go to System Settings > Search > Elasticsearch 7
      4. Confirm that "Production Mode Enabled" is checked and select "test" (or the Connection ID from step 2) within the "Remote Cluster Connection ID" dropdown and save

      Regression was caught by functional test Elasticsearch7#SwitchSidecarToRemote (Testray history) which is also showing the following exception:

      org.openqa.selenium.WebDriverException: unknown error: net::ERR_CONNECTION_REFUSED   (Session info: chrome=86.0.4240.111) Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'cloud-10-0-13-52.lax.liferay.com', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.138-rancher', java.version: '1.8.0_121' Driver info: org.openqa.selenium.chrome.ChromeDriver Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 86.0.4240.111, chrome: {chromedriverVersion: 86.0.4240.22 (398b0743353ff..., userDataDir: /tmp/.com.google.Chrome.bde2zB}, goog:chromeOptions: {debuggerAddress: localhost:38237}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: ignore, webauthn:virtualAuthenticators: true} Session ID: 99365b1d31c715f64795fa2e0b4007d9

      Tested on Tomcat 9.0.43 + MySQL 5.7
      Portal master Bundle: 3d81186d60867919f897756ae9ce187d7d96abcf

        Attachments

          Activity

            People

            Assignee:
            joshua.chong Joshua Chong
            Reporter:
            brooke.dalton Brooke Dalton
            Participants of an Issue:
            Recent user:
            Sophia Zhang
            Engineering Assignee:
            Drew Brokke
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since last comment:
              19 weeks, 1 day ago
              Development End Date:
              Development Start Date:

                Packages

                Version Package
                7.4.2 CE GA3 DXP 7,4
                7.4.13 DXP GA1
                Master