Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-97609

Portal unable to connect to Solr with SSL configuration during initial startup

    Details

      Description

      Steps to reproduce:

      Set up Portal with Solr SSL. Simplest way is to use existing test build scripts at source:

      1. Execute the following to build and deploy Solr jars into your bundle and set up osgi config blacklists for Elasticsearch modules:
        ant -f build-test-solr.xml deploy-solr
        
      2. Prep build scripts to set up Solr SSL configurations by changing this line in build-test-solr.xml to:
        <equals arg1="true" arg2="true" />
        
        1. Note: you can't just pass solr.ssl.authentication.enabled=true in command line because the get-testcase-property call will reset it
      3. Execute the following to download the Solr binary, set up configs and SSL keystores for Solr server / Portal:
        ant -f build-test-solr.xml start-solr
        
      4. Start Portal

      Results: Startup errors indicating Portal is unable to connect to the Solr server. Inability to perform searches in Portal.

      2019-06-28 17:36:43.978 INFO  [CM Event Dispatcher (Fire ConfigurationEvent: pid=com.liferay.portal.search.solr7.configuration.SolrConfiguration)][KeyStoreLoaderImpl:124] Attempting to load from the file system because /keystore.jks is not in the class path
      2019-06-28 17:36:43.982 ERROR [CM Event Dispatcher (Fire ConfigurationEvent: pid=com.liferay.portal.search.solr7.configuration.SolrConfiguration)][SolrClientManager:93] bundle com.liferay.portal.search.solr7.impl:3.0.0 (874)[com.liferay.portal.search.solr7.internal.connection.SolrClientManager(4975)] : The activate method has thrown an exception 
      java.io.FileNotFoundException: /keystore.jks (No such file or directory)
      	at java.io.FileInputStream.open0(Native Method)
      	at java.io.FileInputStream.open(FileInputStream.java:195)
      	at java.io.FileInputStream.<init>(FileInputStream.java:138)
      	at java.io.FileInputStream.<init>(FileInputStream.java:93)
      	at com.liferay.portal.search.solr7.internal.http.KeyStoreLoaderImpl.loadFile(KeyStoreLoaderImpl.java:129)
      	at com.liferay.portal.search.solr7.internal.http.KeyStoreLoaderImpl.load(KeyStoreLoaderImpl.java:53)
      	at com.liferay.portal.search.solr7.internal.http.SSLSocketFactoryBuilderImpl.build(SSLSocketFactoryBuilderImpl.java:52)
      	at com.liferay.portal.search.solr7.internal.http.CertAuthPoolingHttpClientFactory.createPoolingHttpClientConnectionManager(CertAuthPoolingHttpClientFactory.java:78)
      	at com.liferay.portal.search.solr7.internal.http.BasePoolingHttpClientFactory.createInstance(BasePoolingHttpClientFactory.java:46)
      	at com.liferay.portal.search.solr7.internal.connection.ReplicatedSolrClientFactory.getSolrClient(ReplicatedSolrClientFactory.java:70)
      	at com.liferay.portal.search.solr7.internal.connection.SolrClientManager.activate(SolrClientManager.java:93)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:228)
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:664)
      	at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:510)
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:317)
      	at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:307)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeModifiedMethod(SingleComponentManager.java:837)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.modify(SingleComponentManager.java:792)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:710)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.reconfigure(SingleComponentManager.java:674)
      	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.configurationUpdated(ConfigurableComponentHolder.java:435)
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport.configurationEvent(RegionConfigurationSupport.java:317)
      	at org.apache.felix.scr.impl.manager.RegionConfigurationSupport$2.configurationEvent(RegionConfigurationSupport.java:119)
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.sendEvent(ConfigurationManager.java:1685)
      	at org.apache.felix.cm.impl.ConfigurationManager$FireConfigurationEvent.run(ConfigurationManager.java:1637)
      	at org.apache.felix.cm.impl.UpdateThread.run0(UpdateThread.java:138)
      	at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:105)
      	at java.lang.Thread.run(Thread.java:748)
      

      Further along in startup:

      2019-06-28 17:37:44.765 ERROR [liferay/search_writer/SYSTEM_ENGINE-2][SolrIndexWriter:111] org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr
      org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
      	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:663)
      	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:638)
      	at com.liferay.portal.search.solr7.internal.connection.ReadWriteSolrClient.request(ReadWriteSolrClient.java:56)
      	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
      	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
      	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:448)
      	at com.liferay.portal.search.solr7.internal.SolrIndexWriter.commit(SolrIndexWriter.java:107)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:82)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:58)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
      	at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
      	at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.http.client.ClientProtocolException
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
      	... 20 more
      Caused by: org.apache.http.ProtocolException: The server failed to respond with a valid HTTP response
      	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:151)
      	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
      	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
      	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
      	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      	... 23 more
      

       
      Workaround: Restarting Portal resolves the issue. Portal is able to connect to Solr server with no errors.

      QA investigation notes: From the log errors, it is almost as if Portal attempts to connect to Solr before the Solr osgi configs are applied (see example Solr osgi configs attached to this ticket). E.g. Portal tries to read /keystore.jks instead of /home/joshchong/liferay/bundles/solr-snapshot5/solr-7.5.0/server/etc/solr-ssl.keystore.jks, or attempts to connect to http://localhost:8983/solr instead of https://localhost:8983/solr. However, checking system settings after Portal starts up shows that these osgi config settings did eventually get applied. Running RCA tool indicates that this was caused by https://issues.liferay.com/browse/LPS-96739.

      History of the automated test failing because of this: https://testray.liferay.com/home/-/testray/cases/439028385

      Reproduced on:
      Portal master Git SHA: 3601af3fe68f8e4a6238e59124c6d4352e71f8ab
      Portal 7.2.x Git SHA: 71572b9ce6c3d2225c55ffdf0b7b38b5424bb892

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joshua.chong Joshua Chong
                Reporter:
                joshua.chong Joshua Chong
                Participants of an Issue:
                Recent user:
                Jason Pince
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  26 weeks, 3 days ago

                  Packages

                  Version Package
                  7.2.10 DXP FP1
                  7.2.X
                  7.2.1 CE GA2
                  Master