Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-107863

NoNodeAvailableException using Elasticsearch 7 with SSL/TLS on JDK 11 (Transport Client only)

    Details

      Description

      Master/7.3.x is not affected because we are using the REST client for ES7


      Workaround: Add
      xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1" ]
      to elasticsearch.yml of each Elasticsearch node and restart the Elasticsearch cluster.

      Steps to reproduce

      1. Configure Portal 7.2 GA2+/DXP 7.2 SP1/FP2+ with remote Elasticsearch 7 using SSL/TLS (tested on 7.3.1 and 7.5.2)
        1. https://help.liferay.com/hc/en-us/articles/360028711172-Installing-Liferay-Enterprise-Search-Security
        2. https://help.liferay.com/hc/en-us/articles/360035444872-Upgrading-to-Elasticsearch-7
        3. You'll need to deploy the Liferay (CE) Connector to Elasticsearch 7 from Marketplace - Reproduced with Liferay (CE) Connector to Elasticsearch 7 version 3.0.0
      2. Configure both DXP and Elasticsearch to use JDK 11 (Oracle or OpenJDK)
        1. For Elasticsearch 7 JAVA_HOME must be set explicitly to use your own Java otherwise it'll use its bundled JDK (see https://www.elastic.co/blog/elasticsearch-7-0-0-released "Bundled Java")
      3. Start Elasticsearch
      4. Start DXP

      Expected result: connection to remote Elasticsearch 7 cluster over SSL/TLS is successful.
      Actual result: error is thrown, connection fails:

      Liferay log
      2020-01-30 00:31:38.655 ERROR [http-nio-7211-exec-3][ElasticsearchSearchEngineInformation:88] Unable to get node information
      NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{hXkWLVw3RAqhBAYFsQWJGw}{localhost}{127.0.0.1:9300}]]
      	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
      	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
      	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
      	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
      	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392)
      	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:381)
      	at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:675)
      	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:45)
      	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:59)
      	at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation._getClusterNodes(ElasticsearchSearchEngineInformation.java:152)
      	at com.liferay.portal.search.elasticsearch7.internal.information.ElasticsearchSearchEngineInformation.getNodesString(ElasticsearchSearchEngineInformation.java:65)
      

      Analysis, background
      Increasing the SSL log level both for DXP and Elasticsearch by using the -Djavax.net.debug=ssl:handshake:verbose JVM flag reveals the following error during the SSL handshake between DXP (using the Elasticsearch Transport Client) and Elasticsearch node:

       
      javax.net.ssl|ERROR|6E|elasticsearch[es-node1][transport_worker][T#13]|2020-01-24 18:13:26.556 CET|TransportContext.java:313|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (}}{{"throwable" : {}}{{  javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify}}{{      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)}}{{      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)}}{{      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:255)}}{{      at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:737)}}{{      at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1781)}}{{      at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1073)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)}}{{      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)}}{{      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)}}{{      at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913)}}{{      at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)}}{{      at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)}}{{      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)}}{{      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)}}{{      at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)}}{{      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)}}{{      at java.base/java.lang.Thread.run(Thread.java:834)}}}{{)
      
      • Hint: To enabling verbose SSL logging for DXP, you can add CATALINA_OPTS="$CATALINA_OPTS -Djavax.net.debug=ssl:handshake:verbose" to setenv.sh
      • Hint: To enabling verbose SSL logging for Elasticsearch, add -Djavax.net.debug=ssl:handshake:verbose to the end of $ES_HOME/config/jvm.options.

      Related to https://github.com/netty/netty/issues/8463, fixed in 4.1.32.Final or higher.

        Attachments

          Activity

            People

            Assignee:
            brian.chan Brian Chan
            Reporter:
            tibor.lipusz Tibor Lipusz
            Participants of an Issue:
            Recent user:
            Tibor Lipusz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since last comment:
              23 weeks, 1 day ago

                Packages

                Version Package
                7.2.X
                Master