Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-112251

NoNodeAvailableException occurs and Elasticsearch PKI authentication is not working unless the native realm is also enabled in Elasticsearch

    Details

      Description

      Affects

      • Liferay Connector to X-Pack Security [Elastic Stack 6.x] versions 3.0.0, 2.0.0 and 1.0.0, fixed in: TBR*
      • Liferay Connector to Elasticsearch 7 versions 3.0.0, 3.0.1, fixed in: TBR*

      * DXP subscribers can request the fix through Support in a form of a Hotfix LPKG.


      Background

      Authentication and SSL/TLS can be configured independently in Elasticsearch, for example you don't need auth to use SSL/TLS. In addition, deployments using PKI may not want to enable the native realm (username-password) auth. However, currently we create an encrypted connection only if "requiresAuthentication" is enabled in our configs and that requires the username-password thus the native realm to be also enabled in Elasticsearch.


      Steps to Reproduce - Common steps

      1. Download the specified Elasticsearch version and extract to somewhere
      2. Install the necessary analysis plugins:
        ./bin/elasticsearch-plugin install analysis-icu
        ./bin/elasticsearch-plugin install analysis-kuromoji
        ./bin/elasticsearch-plugin install analysis-smartcn
        ./bin/elasticsearch-plugin install analysis-stempel
        
      3. Generate self Certificate Authority (CA) - PEM
            ./bin/elasticsearch-certutil ca --pem
        
        1. Use "liferay" as password if prompted
        2. Copy the files to ES_HOME/config/certs
      4. Generate cert for the ES node - PEM
            ./bin/elasticsearch-certutil cert --ca-cert config/certs/ca.crt --ca-pass liferay --name "CN=liferay.com,OU=Search,DC=liferay,DC=com" --dns localhost --pem --ca-key config/certs/ca.key
        
      5. Rename the generated files as below and copy them to ES_HOME/config/certs:
            elastic-certificates.crt
            elastic-certificates.key
        

      Steps to Reproduce - DXP + Elasticsearch 6.8.x (Applies to all Liferay 7.x versions and branches).

      1. Follow the the common steps above, use for example Elasticsearch 6.8.7
      2. Configure Elasticsearch 6
        1. Configure ES_HOME/config/elasticsearch.yml as
              cluster.name: LiferayElasticsearchCluster
              xpack.security.enabled: true
              xpack.security.transport.ssl.enabled: true
              xpack.security.http.ssl.enabled: true
              xpack.ssl.verification_mode: certificate
              xpack.ssl.key: certs/elastic-certificates.key
              xpack.ssl.certificate: certs/elastic-certificates.crt
              xpack.ssl.certificate_authorities : [ "certs/ca.crt" ]
              xpack.monitoring.collection.enabled: true
              xpack.security.transport.ssl.client_authentication: required
              xpack.security.http.ssl.client_authentication: required
          
        2. Start Elasticsearch
        3. Configure built-in X-Pack users' password
        4. Execute
          ./bin/elasticsearch-setup-passwords interactive
          

          Note: Use "liferay" as password when prompted for consistency

        5. Install a Gold or Platimum ES license or start a free Trial license (Refer to https://www.elastic.co/subscriptions#elastic-stack-security)
          1. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/start-trial.html
        6. Stop Elasticsearch
        7. Enable PKI in ES_HOME/config/elasticsearch.yml:
              xpack.security.authc:
                realms:
                  pki:
                    type: pki
                    order: 0
          
        8. Update ES_HOME/config/role_mapping.yml:
          superuser:
            - "CN=liferay.com,OU=Search,DC=liferay,DC=com"
          
        9. Start Elasticsearch and verify that PKI auth is working:
          1. Go to ES_HOME/config/certs
          2. Execute:
                curl https://localhost:9200/_xpack/security/_authenticate?pretty --key elastic-certificates.key --cert elastic-certificates.crt --cacert ca.crt -v
            

            Note: The response will be something like this:

            *   Trying 127.0.0.1...
            (...)
            * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
            * ALPN, server did not agree to a protocol
            * Server certificate:
            *  subject: DC=com; DC=liferay; OU=Search; CN=liferay.com
            *  start date: Apr 16 19:36:22 2020 GMT
            *  expire date: Apr 16 19:36:22 2023 GMT
            *  subjectAltName: host "localhost" matched cert's "localhost"
            *  issuer: CN=Elastic Certificate Tool Autogenerated CA
            *  SSL certificate verify ok.
            > GET /_xpack/security/_authenticate?pretty HTTP/1.1
            > Host: localhost:9200
            > User-Agent: curl/7.58.0
            > Accept: */*
            > 
            < HTTP/1.1 200 OK
            < content-type: application/json; charset=UTF-8
            < content-length: 352
            < 
            {
              "username" : "liferay.com",
              "roles" : [
                "superuser"
              ],
              "full_name" : null,
              "email" : null,
              "metadata" : {
                "pki_dn" : "CN=liferay.com, OU=Search, DC=liferay, DC=com"
              },
              "enabled" : true,
              "authentication_realm" : {
                "name" : "pki",
                "type" : "pki"
              },
              "lookup_realm" : {
                "name" : "pki",
                "type" : "pki"
              }
            }
            * Connection #0 to host localhost left intact
            

            Note: Notice the JSON with the resolved user and the assigned role.

      3. Configure DXP:
        1. Configure the Elasticsearch 6 connector:
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
          
        2. Configure Connector to X-Pack Security [Elastic Stack 6.x] in DXP:
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch6.xpack.security.internal.configuration.XPackSecurityConfiguration.config
          sslKeyPath="/PATH/TO/config/certs/elastic-certificates.key"
          sslCertificatePath="/PATH/TO/config/certs/elastic-certificates.crt"
          certificateFormat="PEM"
          requiresAuthentication=B"false"
          username=""
          password=""
          sslCertificateAuthoritiesPaths="/PATH/TO/config/certs/ca.crt"
          transportSSLVerificationMode="certificate"
          transportSSLEnabled=B"true"
          

          Note: update the PATHS to reflect your env.

        3. Place Liferay Connector to X-Pack Security [Elastic Stack 6.x].lpkg to LIFERAY_HOME/osgi/marketplace or build & deploy from branch (modules/dxp/apps/portal-search-elasticsearch6-xpack-security)
        4. Start DXP

      Expected Result: DXP is able to connect to Elasticsearch over SSL/TLS.
      Actual Result:

      Error in DXP log:

      Caused by: java.lang.RuntimeException: org.elasticsearch.client.transport.NoNodeAvailableException: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{c19h9RzUSO-_5vBVVjCDgw}{localhost}{127.0.0.1:9300}]]
      

      Error in Elasticsearch 6 log:

      [2020-04-21T16:49:32,330][WARN ][o.e.t.TcpTransport       ] [es-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/127.0.0.1:58978}], closing connection
      io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000053000000000000000308004d3603010f582d466f756e642d436c75737465721b4c696665726179456c6173746963736561726368436c75737465720016696e7465726e616c3a7463702f68616e647368616b6500
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
      	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
      Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000053000000000000000308004d3603010f582d466f756e642d436c75737465721b4c696665726179456c6173746963736561726368436c75737465720016696e7465726e616c3a7463702f68616e647368616b6500
      	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1182) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
      	... 15 more
      

      The error in Elasticsearch indicates that the server (ES) received a plain HTTP request while it is configured to accept SSL/TLS only.


      Steps to reproduce - DXP/Liferay Portal + Elasticsearch 7.x

      1. Follow the common steps, use for example Elasticsearch 7.7.0
      2. Generate cert in PKCS#12 format as well for your ES7 node (required when testing on Master/7.3 - REST Client):
        ./bin/elasticsearch-certutil cert --ca-cert config/certs/ca.crt --ca-pass liferay --name "CN=liferay.com,OU=Search,DC=liferay,DC=com" --dns localhost --ca-key config/certs/ca.key
        
        1. Copy the generated file as elastic-certificates.p12 to ES_HOME/config/certs
      3. Configure Elasticsearch 7:
        1. Configure ES_HOME/config/elasticsearch.yml as
          cluster.name: LiferayElasticsearchCluster
          
          xpack.security.enabled: true
          
          # TLS/SSL settings for Transport layer
          xpack.security.transport.ssl.enabled: true
          xpack.security.transport.ssl.verification_mode: certificate
          xpack.security.transport.ssl.key: certs/elastic-certificates.key
          xpack.security.transport.ssl.certificate: certs/elastic-certificates.crt
          xpack.security.transport.ssl.certificate_authorities : ["certs/ca.crt"]
          xpack.security.transport.ssl.client_authentication: required
          
          # TLS/SSL settings for HTTP layer - required when testing on master/7.3.
          xpack.security.http.ssl.enabled: true
          xpack.security.http.ssl.verification_mode: certificate
          xpack.security.http.ssl.key: certs/elastic-certificates.key
          xpack.security.http.ssl.certificate: certs/elastic-certificates.crt
          xpack.security.http.ssl.certificate_authorities : ["certs/ca.crt"]
          
        2. Start ES
        3. Configure built-in X-Pack users' password
        4. Execute
          ./bin/elasticsearch-setup-passwords interactive
          

          Note: Use "liferay" as password when prompted for consistency

        5. Install a Gold or Platimum ES license or start a free Trial license (Refer to https://www.elastic.co/subscriptions#elastic-stack-security)
          1. https://www.elastic.co/guide/en/elasticsearch/reference/7.x/start-trial.html
        6. Stop Elasticsearch
        7. Configure PKI: add the following to the end of ES_HOME/config/elasticsearch.yml:
          xpack:
            security:
              authc:
                realms:
                  pki:
                    pki1:
                      order: 0
          
        8. Update ES_HOME/config/role_mapping.yml:
          superuser:
            - "CN=liferay.com,OU=Search,DC=liferay,DC=com"
          
      4. Configure Master/7.3/7.2:
        1. Blacklist the default ES6 connector:
          LIFERAY_HOME/config/com.liferay.portal.bundle.blacklist.internal.BundleBlacklistConfiguration.config
          blacklistBundleSymbolicNames=[ \
          	"com.liferay.portal.search.elasticsearch6.api", \
          	"com.liferay.portal.search.elasticsearch6.impl", \
          	"com.liferay.portal.search.elasticsearch6.spi", \
          	"com.liferay.portal.search.elasticsearch6.xpack.security.impl", \
          	"Liferay Connector to X-Pack Security [Elastic Stack 6.x] - Impl" \
          ]
          
        2. Configure the Elasticsearch 7 connector - 7.2/7.2.x (Transport Client):
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
          
          1. Configure X-Pack Security in the Elasticsearch 7 connector:
            LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.XPackSecurityConfiguration.config
            sslKeyPath="/PATH/TO/config/certs/elastic-certificates.key"
            sslCertificatePath="/PATH/TO/config/certs/elastic-certificates.crt"
            certificateFormat="PEM"
            requiresAuthentication=B"false"
            username=""
            password=""
            sslCertificateAuthoritiesPaths="/PATH/TO/config/certs/ca.crt"
            transportSSLVerificationMode="certificate"
            transportSSLEnabled=B"true"
            

            Note: update the PATHS to reflect your env.

          2. Deploy the latest version of the Liferay Connector to Elasticsearch 7 for your portal version from Marketplace or build it from the branch (modules/apps/portal-search-elasticsearch7)
        3. Configure the Elasticsearch 7 connector - Master/7.3 (REST client):
          1. Build portal-search-elasticsearch7 from master
          2. When you build elasticsearch7, it creates a "-default" config file in "osgi/configs". Replace its content with settings below:
            LIFERAY_HOME/osgi/configs/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConnectionConfiguration-default.config
            active=B"true"
            authenticationEnabled=B"false"
            connectionId="remote"
            httpSSLEnabled=B"true"
            networkHostAddresses=["https://localhost:9200"]
            truststorePassword="liferay"
            truststorePath="/PATH/TO/certs/elastic-certificates.p12"
            truststoreType="pkcs12"
            

            Note: don't forget to update the path to match your env.

        4. Create the following config:
          LIFERAY_HOME/osgi/configs/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
          remoteClusterConnectionId="remote"
          
      5. Start ES
      6. Start portal

      The error will be the same as with ES6. You can also verify that PKI authentication works by executing the curl command listed in step 2.8 in the Elasticsearch 6 steps.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brooke.dalton Brooke Dalton
                Reporter:
                tibor.lipusz Tibor Lipusz
                Participants of an Issue:
                Recent user:
                Jason Pince
                Engineering Assignee:
                Bryan Engler
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  11 weeks, 5 days ago

                  Packages

                  Version Package
                  7.0.X
                  7.1.X
                  7.2.X
                  7.3.3 CE GA4
                  Master