NoNodeAvailableException using Elasticsearch 7 with SSL/TLS on JDK 11 (Transport Client only)

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.2.10.1 DXP SP1, 7.2.1 CE GA2, 7.2.X
• Fix Version/s: 7.2.X, Master
• Component/s: Search Infrastructure > Elasticsearch Connector
• Labels:
  • ee-bp
  • ee-ts
  • marketplace-portal-search-elasticsearch7-3.0.1-7210
• Branch Version/s:
  7.2.x
• Backported to Branch:
  Committed
• Fix Priority: 4
• JDK:
  Oracle Sun JDK 11, Oracle OpenJDK 11
• Sprint: Search|S03 Sprint 11 |2.3-2.21
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/84202

Master/7.3.x is not affected because we are using the REST client for ES7

---

Workaround: Add
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1" ]
to elasticsearch.yml of each Elasticsearch node and restart the Elasticsearch cluster.

Steps to reproduce

1. Configure Portal 7.2 GA2+/DXP 7.2 SP1/FP2+ with remote Elasticsearch 7 using SSL/TLS (tested on 7.3.1 and 7.5.2)
   1. https://help.liferay.com/hc/en-us/articles/360028711172-Installing-Liferay-Enterprise-Search-Security
   2. https://help.liferay.com/hc/en-us/articles/360035444872-Upgrading-to-Elasticsearch-7
   3. You'll need to deploy the Liferay (CE) Connector to Elasticsearch 7 from Marketplace - Reproduced with Liferay (CE) Connector to Elasticsearch 7 version 3.0.0
2. Configure both DXP and Elasticsearch to use JDK 11 (Oracle or OpenJDK)
   1. For Elasticsearch 7 JAVA_HOME must be set explicitly to use your own Java otherwise it'll use its bundled JDK (see https://www.elastic.co/blog/elasticsearch-7-0-0-released "Bundled Java")
3. Start Elasticsearch
4. Start DXP

Expected result: connection to remote Elasticsearch 7 cluster over SSL/TLS is successful.
Actual result: error is thrown, connection fails:

Liferay log

2020-01-30 00:31:38.655 ERROR [http-nio-7211-exec-3][ElasticsearchSearchEngineInformation:88] Unable to get node information
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{hXkWLVw3RAqhBAYFsQWJGw}{localhost}{127.0.0.1:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:381)
	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:675)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45)
	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:59)
	at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation._getClusterNodes(ElasticsearchSearchEngineInformation.java:152)
	at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation.getNodesString(ElasticsearchSearchEngineInformation.java:65)

---

Analysis, background
Increasing the SSL log level both for DXP and Elasticsearch by using the -Djavax.net.debug=ssl:handshake:verbose JVM flag reveals the following error during the SSL handshake between DXP (using the Elasticsearch Transport Client) and Elasticsearch node:

 
javax.net.ssl|ERROR|6E|elasticsearch[es-node1][transport_worker][T#13]|2020-01-24 18:13:26.556 CET|TransportContext.java:313|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (}}{{"throwable" : {}}{{  javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify}}{{      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)}}{{      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:255)}}{{      at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:737)}}{{      at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1781)}}{{      at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1073)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)}}{{      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{      at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913)}}{{      at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)}}{{      at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)}}{{      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)}}{{      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)}}{{      at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)}}{{      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)}}{{      at java.base/java.lang.Thread.run(Thread.java:834)}}}{{)

• Hint: To enabling verbose SSL logging for DXP, you can add CATALINA_OPTS="$CATALINA_OPTS -Djavax.net.debug=ssl:handshake:verbose" to setenv.sh
• Hint: To enabling verbose SSL logging for Elasticsearch, add -Djavax.net.debug=ssl:handshake:verbose to the end of $ES_HOME/config/jvm.options.

Related to https://github.com/netty/netty/issues/8463, fixed in 4.1.32.Final or higher.