-
Type:
Bug
-
Status: Closed
-
Resolution: Fixed
-
Affects Version/s: 7.2.10.1 DXP SP1, 7.2.X, 7.2.1 CE GA2
-
Component/s: Search Infrastructure > Elasticsearch Connector
-
Branch Version/s:7.2.x
-
Backported to Branch:Committed
-
Fix Priority:4
-
JDK:Oracle Sun JDK 11, Oracle OpenJDK 11
-
Sprint:Search|S03 Sprint 11 |2.3-2.21
-
Git Pull Request:
Master/7.3.x is not affected because we are using the REST client for ES7
Workaround: Add
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1" ]
to elasticsearch.yml of each Elasticsearch node and restart the Elasticsearch cluster.
Steps to reproduce
- Configure Portal 7.2 GA2+/DXP 7.2 SP1/FP2+ with remote Elasticsearch 7 using SSL/TLS (tested on 7.3.1 and 7.5.2)
- https://help.liferay.com/hc/en-us/articles/360028711172-Installing-Liferay-Enterprise-Search-Security
- https://help.liferay.com/hc/en-us/articles/360035444872-Upgrading-to-Elasticsearch-7
- You'll need to deploy the Liferay (CE) Connector to Elasticsearch 7 from Marketplace -
Reproduced with Liferay (CE) Connector to Elasticsearch 7 version 3.0.0
- Configure both DXP and Elasticsearch to use JDK 11 (Oracle or OpenJDK)
- For Elasticsearch 7 JAVA_HOME must be set explicitly to use your own Java otherwise it'll use its bundled JDK (see https://www.elastic.co/blog/elasticsearch-7-0-0-released "Bundled Java")
- Start Elasticsearch
- Start DXP
Expected result: connection to remote Elasticsearch 7 cluster over SSL/TLS is successful.
Actual result: error is thrown, connection fails:
2020-01-30 00:31:38.655 ERROR [http-nio-7211-exec-3][ElasticsearchSearchEngineInformation:88] Unable to get node information NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{hXkWLVw3RAqhBAYFsQWJGw}{localhost}{127.0.0.1:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:381) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:675) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45) at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:59) at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation._getClusterNodes(ElasticsearchSearchEngineInformation.java:152) at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation.getNodesString(ElasticsearchSearchEngineInformation.java:65)
Analysis, background
Increasing the SSL log level both for DXP and Elasticsearch by using the -Djavax.net.debug=ssl:handshake:verbose JVM flag reveals the following error during the SSL handshake between DXP (using the Elasticsearch Transport Client) and Elasticsearch node:
javax.net.ssl|ERROR|6E|elasticsearch[es-node1][transport_worker][T#13]|2020-01-24 18:13:26.556 CET|TransportContext.java:313|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (}}{{"throwable" : {}}{{ javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify}}{{ at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)}}{{ at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)}}{{ at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)}}{{ at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)}}{{ at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:255)}}{{ at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:737)}}{{ at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1781)}}{{ at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1073)}}{{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{ at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)}}{{ at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417)}}{{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{ at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913)}}{{ at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)}}{{ at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)}}{{ at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)}}{{ at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)}}{{ at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)}}{{ at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)}}{{ at java.base/java.lang.Thread.run(Thread.java:834)}}}{{)
- Hint: To enabling verbose SSL logging for DXP, you can add CATALINA_OPTS="$CATALINA_OPTS -Djavax.net.debug=ssl:handshake:verbose" to setenv.sh
- Hint: To enabling verbose SSL logging for Elasticsearch, add -Djavax.net.debug=ssl:handshake:verbose to the end of $ES_HOME/config/jvm.options.
Related to https://github.com/netty/netty/issues/8463, fixed in 4.1.32.Final or higher.