Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-112251

NoNodeAvailableException occurs and Elasticsearch PKI authentication is not working unless the native realm is also enabled in Elasticsearch

    XMLWordPrintable

Details

    Description

      Affects

      • Liferay Enterprise Search Security/Connector to X-Pack Security [Elastic Stack 6.x] versions 3.0.0, 2.0.0 and 1.0.0, fixed in: TBR*
      • Liferay Connector to Elasticsearch 7 versions 3.0.0, 3.0.1, fixed in: 3.1.0+ (for DXP 7.2), 2.0.0 (for DXP 7.1)

      * DXP subscribers can request the fix through Support in a form of a Hotfix LPKG.

      Background

      Authentication and SSL/TLS can be configured independently in Elasticsearch, for example you don't need auth to use SSL/TLS. In addition, deployments using PKI may not want to enable the native realm (username-password) auth. However, currently we create an encrypted connection only if "requiresAuthentication" is enabled in our configs and that requires the username-password thus the native realm to be also enabled in Elasticsearch.

      Steps to Reproduce - Common steps

      1. Download the specified Elasticsearch version and extract to somewhere
      2. Install the necessary analysis plugins: 
        ./bin/elasticsearch-plugin install analysis-icu
./bin/elasticsearch-plugin install analysis-kuromoji
./bin/elasticsearch-plugin install analysis-smartcn
./bin/elasticsearch-plugin install analysis-stempel
      3. Generate self Certificate Authority (CA) - PEM 
            ./bin/elasticsearch-certutil ca --pem
        1. Use "liferay" as password if prompted
        2. Copy the files to ES_HOME/config/certs
      4. Generate cert for the ES node - PEM 
            ./bin/elasticsearch-certutil cert --ca-cert config/certs/ca.crt --ca-pass liferay --name "CN=liferay.com,OU=Search,DC=liferay,DC=com" --dns localhost --pem --ca-key config/certs/ca.key
      5. Rename the generated files as below and copy them to ES_HOME/config/certs: 
            elastic-certificates.crt
    elastic-certificates.key

      Steps to Reproduce - DXP + Elasticsearch 6.8.x (Applies to all Liferay 7.x versions and branches).

      1. Follow the the common steps above, use for example Elasticsearch 6.8.7
      2. Configure Elasticsearch 6
        1. Configure ES_HOME/config/elasticsearch.yml as 
              cluster.name: LiferayElasticsearchCluster
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.ssl.verification_mode: certificate
    xpack.ssl.key: certs/elastic-certificates.key
    xpack.ssl.certificate: certs/elastic-certificates.crt
    xpack.ssl.certificate_authorities : [ "certs/ca.crt" ]
    xpack.monitoring.collection.enabled: true
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.http.ssl.client_authentication: required
        2. Start Elasticsearch
        3. Configure built-in X-Pack users' password
        4. Execute 
          ./bin/elasticsearch-setup-passwords interactive

          Note: Use "liferay" as password when prompted for consistency

        5. Install a Gold or Platimum ES license or start a free Trial license (Refer to https://www.elastic.co/subscriptions#elastic-stack-security)
          1. https://www.elastic.co/guide/en/elasticsearch/reference/6.8/start-trial.html
        6. Stop Elasticsearch
        7. Enable PKI in ES_HOME/config/elasticsearch.yml: 
              xpack.security.authc:
      realms:
        pki:
          type: pki
          order: 0
        8. Update ES_HOME/config/role_mapping.yml: 
          superuser:
  - "CN=liferay.com,OU=Search,DC=liferay,DC=com"
        9. Start Elasticsearch and verify that PKI auth is working:
          1. Go to ES_HOME/config/certs
          2. Execute: 
                curl https://localhost:9200/_xpack/security/_authenticate?pretty --key elastic-certificates.key --cert elastic-certificates.crt --cacert ca.crt -v

            Note: The response will be something like this:

            *   Trying 127.0.0.1...
(...)
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=com; DC=liferay; OU=Search; CN=liferay.com
*  start date: Apr 16 19:36:22 2020 GMT
*  expire date: Apr 16 19:36:22 2023 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=Elastic Certificate Tool Autogenerated CA
*  SSL certificate verify ok.
> GET /_xpack/security/_authenticate?pretty HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 352
< 
{
  "username" : "liferay.com",
  "roles" : [
    "superuser"
  ],
  "full_name" : null,
  "email" : null,
  "metadata" : {
    "pki_dn" : "CN=liferay.com, OU=Search, DC=liferay, DC=com"
  },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "pki",
    "type" : "pki"
  },
  "lookup_realm" : {
    "name" : "pki",
    "type" : "pki"
  }
}
* Connection #0 to host localhost left intact

            Note: Notice the JSON with the resolved user and the assigned role.

      3. Configure DXP:
        1. Configure the Elasticsearch 6 connector:
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch6.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
        2. Configure Connector to X-Pack Security [Elastic Stack 6.x] in DXP:
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch6.xpack.security.internal.configuration.XPackSecurityConfiguration.config
          sslKeyPath="/PATH/TO/config/certs/elastic-certificates.key"
sslCertificatePath="/PATH/TO/config/certs/elastic-certificates.crt"
certificateFormat="PEM"
requiresAuthentication=B"false"
username=""
password=""
sslCertificateAuthoritiesPaths="/PATH/TO/config/certs/ca.crt"
transportSSLVerificationMode="certificate"
transportSSLEnabled=B"true"

          Note: update the PATHS to reflect your env.

        3. Place Liferay Connector to X-Pack Security [Elastic Stack 6.x].lpkg to LIFERAY_HOME/osgi/marketplace or build & deploy from branch (modules/dxp/apps/portal-search-elasticsearch6-xpack-security)
        4. Start DXP

      Expected Result: DXP is able to connect to Elasticsearch over SSL/TLS.
      Actual Result:

      Error in DXP log:

      Caused by: java.lang.RuntimeException: org.elasticsearch.client.transport.NoNodeAvailableException: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{c19h9RzUSO-_5vBVVjCDgw}{localhost}{127.0.0.1:9300}]]

      Error in Elasticsearch 6 log:

      [2020-04-21T16:49:32,330][WARN ][o.e.t.TcpTransport       ] [es-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/127.0.0.1:58978}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000053000000000000000308004d3603010f582d466f756e642d436c75737465721b4c696665726179456c6173746963736561726368436c75737465720016696e7465726e616c3a7463702f68616e647368616b6500
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000053000000000000000308004d3603010f582d466f756e642d436c75737465721b4c696665726179456c6173746963736561726368436c75737465720016696e7465726e616c3a7463702f68616e647368616b6500
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1182) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more

      The error in Elasticsearch indicates that the server (ES) received a plain HTTP request while it is configured to accept SSL/TLS only.

      Steps to reproduce - DXP/Liferay Portal + Elasticsearch 7.x

      1. Follow the common steps, use for example Elasticsearch 7.7.0
      2. Generate cert in PKCS#12 format as well for your ES7 node (required when testing on Master/7.3 - REST Client): 
        ./bin/elasticsearch-certutil cert --ca-cert config/certs/ca.crt --ca-pass liferay --name "CN=liferay.com,OU=Search,DC=liferay,DC=com" --dns localhost --ca-key config/certs/ca.key
        1. Copy the generated file as elastic-certificates.p12 to ES_HOME/config/certs
      3. Configure Elasticsearch 7:
        1. Configure ES_HOME/config/elasticsearch.yml as 
          cluster.name: LiferayElasticsearchCluster

xpack.security.enabled: true

# TLS/SSL settings for Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: certs/elastic-certificates.key
xpack.security.transport.ssl.certificate: certs/elastic-certificates.crt
xpack.security.transport.ssl.certificate_authorities : ["certs/ca.crt"]
xpack.security.transport.ssl.client_authentication: required

# TLS/SSL settings for HTTP layer - required when testing on master/7.3.
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: certs/elastic-certificates.key
xpack.security.http.ssl.certificate: certs/elastic-certificates.crt
xpack.security.http.ssl.certificate_authorities : ["certs/ca.crt"]
        2. Start ES
        3. Configure built-in X-Pack users' password
        4. Execute 
          ./bin/elasticsearch-setup-passwords interactive

          Note: Use "liferay" as password when prompted for consistency

        5. Install a Gold or Platimum ES license or start a free Trial license (Refer to https://www.elastic.co/subscriptions#elastic-stack-security)
          1. https://www.elastic.co/guide/en/elasticsearch/reference/7.x/start-trial.html
        6. Stop Elasticsearch
        7. Configure PKI: add the following to the end of ES_HOME/config/elasticsearch.yml: 
          xpack:
  security:
    authc:
      realms:
        pki:
          pki1:
            order: 0
        8. Update ES_HOME/config/role_mapping.yml: 
          superuser:
  - "CN=liferay.com,OU=Search,DC=liferay,DC=com"
      4. Configure Master/7.3/7.2:
        1. Blacklist the default ES6 connector:
          LIFERAY_HOME/config/com.liferay.portal.bundle.blacklist.internal.BundleBlacklistConfiguration.config
          blacklistBundleSymbolicNames=[ \
	"com.liferay.portal.search.elasticsearch6.api", \
	"com.liferay.portal.search.elasticsearch6.impl", \
	"com.liferay.portal.search.elasticsearch6.spi", \
	"com.liferay.portal.search.elasticsearch6.xpack.security.impl", \
	"Liferay Connector to X-Pack Security [Elastic Stack 6.x] - Impl" \
]
        2. Configure the Elasticsearch 7 connector - 7.2/7.2.x (Transport Client):
          LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
          1. Configure X-Pack Security in the Elasticsearch 7 connector:
            LIFERAY_HOME/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.XPackSecurityConfiguration.config
            sslKeyPath="/PATH/TO/config/certs/elastic-certificates.key"
sslCertificatePath="/PATH/TO/config/certs/elastic-certificates.crt"
certificateFormat="PEM"
requiresAuthentication=B"false"
username=""
password=""
sslCertificateAuthoritiesPaths="/PATH/TO/config/certs/ca.crt"
transportSSLVerificationMode="certificate"
transportSSLEnabled=B"true"

            Note: update the PATHS to reflect your env.

          2. Deploy the latest version of the Liferay Connector to Elasticsearch 7 for your portal version from Marketplace or build it from the branch (modules/apps/portal-search-elasticsearch7)
        3. Configure the Elasticsearch 7 connector - Master/7.3 (REST client):
          1. Build portal-search-elasticsearch7 from master
          2. When you build elasticsearch7, it creates a "-default" config file in "osgi/configs". Replace its content with settings below:
            LIFERAY_HOME/osgi/configs/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConnectionConfiguration-default.config
            active=B"true"
authenticationEnabled=B"false"
connectionId="remote"
httpSSLEnabled=B"true"
networkHostAddresses=["https://localhost:9200"]
truststorePassword="liferay"
truststorePath="/PATH/TO/certs/elastic-certificates.p12"
truststoreType="pkcs12"

            Note: don't forget to update the path to match your env.

        4. Create the following config:
          LIFERAY_HOME/osgi/configs/osgi/configs/com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config
          operationMode="REMOTE"
remoteClusterConnectionId="remote"
      5. Start ES
      6. Start portal

      The error will be the same as with ES6. You can also verify that PKI authentication works by executing the curl command listed in step 2.8 in the Elasticsearch 6 steps.

      Attachments

        Issue Links

          is a dependency of

          Bug - A problem which impairs or prevents the functions of the product. LPS-114511 [CCR] DXP is unable to establish connection to the follower cluster when PKI is configured unless the native realm is also enabled in Elasticsearch

          • Closed

          Spike - LPS-96667 REST client: PKI and other potential security features under subscription

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              brooke.dalton Brooke Dalton
              tibor.lipusz Tibor Lipusz
              Kiyoshi Lee Kiyoshi Lee
              Bryan Engler Bryan Engler
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                3 years, 3 weeks, 2 days ago

                Packages

                  Version Package
                  7.0.X
                  7.1.X
                  7.2.X
                  7.3.3 CE GA4
                  7.3.10 DXP GA1
                  Master