Deploying the CCR module and the ElasticsearchConnectionConfiguration osgi config at the same time breaks CCR

Fix Priority

5

Affects versions

Fix versions

None

Labels

Description

Steps to reproduce:

  1. Use a Portal bundle that has not deployed the CCR module yet

  2. Shut down Portal if it's already running

  3. Prep Portal with CCR (deploy Elasticsearch 7 connector, blacklist Elasticsearch 6,  start two Elasticsearch 7 clusters, enable REMOTE mode, etc see documentation)

  4. Place the CCR module within liferay.home/deploy

  5. Configure the com.liferay.portal.search.elasticsearch.cross.cluster.replication.internal.configuration.ElasticsearchConnectionConfiguration-<connection.name>.config file and place it within liferay.home/osgi/configs

  6. Start Portal

Actual result: The following error is thrown during Portal start up.

2021-02-11 22:08:45.402 ERROR [main][CCRElasticsearchConnection:93] bundle com.liferay.portal.search.elasticsearch7.impl:4.0.10 (207)[com.liferay.portal.search.elasticsearch7.internal.connection.CCRElasticsearchConnection(386)] : The activate method has thrown an exception java.lang.NullPointerException at com.liferay.portal.search.elasticsearch7.internal.connection.CCRElasticsearchConnection.loadRequiredDefaultConfigurations(CCRElasticsearchConnection.java:251) at com.liferay.portal.search.elasticsearch7.internal.connection.BaseElasticsearchConnection.connect(BaseElasticsearchConnection.java:61) at com.liferay.portal.search.elasticsearch7.internal.connection.CCRElasticsearchConnection.reconnect(CCRElasticsearchConnection.java:280) at com.liferay.portal.search.elasticsearch7.internal.connection.CCRElasticsearchConnection.activate(CCRElasticsearchConnection.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:228) at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41) at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:664) at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:510) at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:317) at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:307) at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:341) at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:114) at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:983) at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:956) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:756) at org.apache.felix.scr.impl.manager.DependencyManager$SingleDynamicCustomizer.addedService(DependencyManager.java:833) at org.apache.felix.scr.impl.manager.DependencyManager$SingleDynamicCustomizer.addedService(DependencyManager.java:775) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1216) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1137) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:944) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:880) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1168) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:125) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:891) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:469) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:906) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:892) at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:128) at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:959) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:732) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1053) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1007) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1216) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1137) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:944) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:880) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1168) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:125) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:891) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:469) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:906) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:892) at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:128) at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:959) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:732) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1053) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1007) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1216) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1137) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:944) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:880) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1168) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:125) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:891) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:469) at com.liferay.portal.bootstrap.ModuleFrameworkImpl._registerService(ModuleFrameworkImpl.java:1564) at com.liferay.portal.bootstrap.ModuleFrameworkImpl.lambda$_registerApplicationContext$4(ModuleFrameworkImpl.java:1517) at java.util.Iterator.forEachRemaining(Iterator.java:116) at com.liferay.portal.bootstrap.ModuleFrameworkImpl._registerApplicationContext(ModuleFrameworkImpl.java:1502) at com.liferay.portal.bootstrap.ModuleFrameworkImpl.registerContext(ModuleFrameworkImpl.java:323) at com.liferay.portal.module.framework.ModuleFrameworkUtilAdapter.registerContext(ModuleFrameworkUtilAdapter.java:72) at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:352) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5177) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:631) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1830) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:425) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)

Steps continued:

  1. Navigate to Control Panel > Configuration > Search > Connections
    Actual result: The connection defined within <connection.name> is missing

  2. Navigate to System Settings > Search > Cross-Cluster Replication

  3. Enable Read from Local Clusters

  4. Within the Local Cluster Configurations field enter "localhost:8080,<connection.name>"

  5. Save

Actual Result: The settings are saved and no errors are thrown. However, search functionality across Portal stops working e.g. search widgets become unavailable, the Connections tab in Search Admin becomes unavailable.

Note: Before enabling CCR in System Settings (or disabling CCR after enabling), some CCR features appear to be working. i.e. The CCR specific System Settings are present and the Connections tab appears in Search Admin. Even the <connection.name> connection that is defined in the ElasticsearchConnectionConfiguration osgi config appears within System Settings.


Alternative steps to reproduce:

  1. Start Portal without the CCR module

  2. Deploy the ElasticsearchConnectionConfiguration osgi config file either before or after Portal starts

  3. Deploy the CCR module

Actual results: Same as above.


Workaround options to prevent the issue: 

  1. Only deploy the ElasticsearchConnectionConfiguration osgi config file after Portal starts up and the CCR module is fully activated. i.e. Deploy the CCR module first, then deploy the osgi config, don't do it at the same time.

  2. Don't use the ElasticsearchConnectionConfiguration osgi config. Instead, set up the local connection within Portal in System Settings > Search > Cross-Cluster Replication Elasticsearch Connections

  3. If Portal hasn't started yet, don't "hot" deploy the CCR module at all. i.e. put the module / LPKG in the appropriate locations within liferay.home/osgi instead of liferay.home/deploy

Workaround options after encountering the issue: 

  1. Restart Portal. This seems to resolve all issues previously mentioned. However, only restarting the CCR module and / or ES7 connector modules does not seem to fix the issue

  2. Taken from Bryan's comment. Create a new connection within System Settings > Search > Cross-Cluster Replication Elasticsearch Connections using all of the same settings as the original connection defined in the ElasticsearchConnectionConfiguration osgi config except use a different connection ID. Use this new connection for CCR going forward. This method is preferred if a Portal restart is not feasible.

  3. To bring back search functionality (but without CCR), simply disable CCR in System Settings.

Reproducible on:
DXP 7.2 FP10 + Liferay Enterprise Search Cross-Cluster Replication (3.0.0) + Liferay Connector to Elasticsearch 7 (3.2.1)
Portal 7.2.x git commit 0ac43eb4964e003ac337558eca26717ec705f66e
Portal 7.1.x git commit 8376cd20e64a7e8f52599b8818fd63e119cdcf35

Not Reproducible on:
Portal master git commit 85f66220f377144c4cf025be77a04c763a9f04b2
Portal 7.3.x git commit a40bee78231c97d66bad1c3b6e79fbb2228eef9d
DXP 7.3 GA1 + Liferay Enterprise Search Cross-Cluster Replication (4.0.0)

Activity

Show:

Bryan Engler February 12, 2021 at 11:30 AM

since the steps to reproduce arent fully a valid use case (they are basically trying to create a ccr config without the ccr module, which would be impossible to do via UI), i am going to close this ticket as wont fix but add some documentation with a warning and also a workaround in the event that it does happen.

link to the current documentation pr - https://github.com/jrhoun/liferay-learn/pull/898

Bryan Engler February 12, 2021 at 10:13 AM
Edited

The easiest workaround i found was to just create a new entry in Cross-Cluster Replication Elasticsearch Connections, call it "ccr2" and duplicate all of the settings from "ccr". then use the ccr2 connection in the Local Cluster Configurations.

because the connections logic depends so much on the Config Admin infrastructure, i dont think theres really anything we can do in the elastic or ccr connectors to fix this. the "fix" would be to document that you shouldnt deploy a com.liferay.portal.search.elasticsearch.cross.cluster.replication.internal.configuration.ElasticsearchConnectionConfiguration-ccr.config file if the ccr module is not already activated 

Won't Fix
Pinned fields
Click on the next to a field label to start pinning.

Assignee

SE Support

Reporter

Sprint

Created February 11, 2021 at 4:36 PM
Updated February 21, 2022 at 8:09 AM
Resolved February 12, 2021 at 11:30 AM

Flag notifications