Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-56035

High CPU consumption with Lucene search

    Details

      Description

      When the Lucene index is used to search (i.e. Asset Publisher), and a profile of Liferay Portal is done, a great amount of CPU is used in com.liferay.portal.search.lucene.highlight.QueryTermExtractor.getTerms (see attached image: profile.png)

      If thread dumps are taken in those moments when CPU use is high, a lot of threads are running at this point:

      "http-/10.187.162.42:8081-362" daemon prio=10 tid=0x00007fbd24bd0800 nid=0x3e48 runnable [0x00007fbbc2fa5000]
         java.lang.Thread.State: RUNNABLE
      	at com.liferay.portal.search.lucene.highlight.QueryTermExtractor.getTerms(QueryTermExtractor.java:41)
      	at com.liferay.portal.search.lucene.LuceneHelperImpl.countScoredFieldNames(LuceneHelperImpl.java:314)
      	at com.liferay.portal.search.lucene.LuceneHelperUtil.countScoredFieldNames(LuceneHelperUtil.java:296)
      	at com.liferay.portal.search.lucene.LuceneIndexSearcher.toHits(LuceneIndexSearcher.java:621)
      	at com.liferay.portal.search.lucene.LuceneIndexSearcher.search(LuceneIndexSearcher.java:246)
      	at sun.reflect.GeneratedMethodAccessor1395.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:85)
      	at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:51)
      	at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:72)
      	at com.liferay.portal.kernel.messaging.SynchronousDestination.send(SynchronousDestination.java:41)
      	at com.liferay.portal.kernel.messaging.sender.DirectSynchronousMessageSender.send(DirectSynchronousMessageSender.java:54)
      	at com.liferay.portal.kernel.messaging.proxy.BaseMultiDestinationProxyBean.synchronousSend(BaseMultiDestinationProxyBean.java:48)
      	at com.liferay.portal.messaging.proxy.MultiDestinationMessagingProxyInvocationHandler.invoke(MultiDestinationMessagingProxyInvocationHandler.java:50)
      	at com.sun.proxy.$Proxy288.search(Unknown Source)
      	at com.liferay.portal.kernel.search.SearchEngineUtil.search(SearchEngineUtil.java:671)
      	at com.liferay.portal.kernel.search.BaseIndexer.doSearch(BaseIndexer.java:1364)
      	at com.liferay.portal.kernel.search.DefaultSearchResultPermissionFilter.getHits(DefaultSearchResultPermissionFilter.java:94)
      	at com.liferay.portal.kernel.search.BaseSearchResultPermissionFilter.search(BaseSearchResultPermissionFilter.java:70)
      	at com.liferay.portal.kernel.search.BaseIndexer.search(BaseIndexer.java:527)
      	at com.liferay.portlet.asset.util.AssetUtil.search(AssetUtil.java:626)
      	at com.liferay.portlet.asset.util.AssetUtil.search(AssetUtil.java:571)
      	at org.apache.jsp.html.portlet.asset_005fpublisher.view_jsp._jspService(view_jsp.java:1931)
      

      Steps to reproduce:

      1. Start a clean 6.2.x bundle (tested using these JVM options: -Xmx1624m -XX:MaxPermSize=512m).
      2. As an admin, execute the attached groovy script (Create_contents.groovy) changing the groupId from the script to the groupId of your default site from Control Panel.
      3. When script is completed, add a new page named "ap".
      4. Add an Asset Publisher in "ap" page.
      5. Download JMeter (tested with 2.13) and extract in some directory.
      6. Download JMeter Standard Set (downloaded here: http://jmeter-plugins.org/downloads/file/JMeterPlugins-Standard-1.3.0.zip) and install in previous installed JMeter.
      7. Download JMeter ServerAgent from http://jmeter-plugins.org/downloads/file/ServerAgent-2.2.1.zip. Extract compressed to any directory and execute "bin/startAgent.bat" or "startAgent.sh" depending on Windows/Linux environment.
      8. Open JMeter, and open the attached file (LPS-56035_test.jmx).
      9. Execute test plan clicking on play button in toolbar. Wait until it ends (that should happen when "Stop" button gets disabled).
      10. In the test plan tree (left), click on "Responses time over time". Save the chart (optionally save the CSV to compare using a spreadsheet). Optional: execute the test plan more times for getting more accurate measures to compare. Optional: save the "Perfmon metrics collector" to get CPU measures.
      11. Shutdown Liferay and start up with this fix applied. After the fix, redo steps from 8 to 10 and save the targeted graphs/CSV
      12. Compare the results from before and after the fix.

      After the fix the Response time over time results in "Ap request" are noticeable lower (14,7% better performance than before).

        Attachments

        1. CPU_after1.png
          CPU_after1.png
          43 kB
        2. CPU_after2.png
          CPU_after2.png
          43 kB
        3. CPU_after3.png
          CPU_after3.png
          33 kB
        4. CPU_before1.png
          CPU_before1.png
          36 kB
        5. CPU_before2.png
          CPU_before2.png
          35 kB
        6. CPU_before3.png
          CPU_before3.png
          32 kB
        7. Create_contents.groovy
          4 kB
        8. fixed_CPU.png
          fixed_CPU.png
          30 kB
        9. fixed_ResponseTimesOverTime.png
          fixed_ResponseTimesOverTime.png
          63 kB
        10. LPS-56035_test.jmx
          18 kB
        11. profile.png
          profile.png
          269 kB
        12. reproduced_CPU.png
          reproduced_CPU.png
          30 kB
        13. reproduced_ResponseTimesOverTime.png
          reproduced_ResponseTimesOverTime.png
          63 kB
        14. Response_Time_Graph_after1.png
          Response_Time_Graph_after1.png
          76 kB
        15. Response_Time_Graph_after2.png
          Response_Time_Graph_after2.png
          73 kB
        16. Response_Time_Graph_after3.png
          Response_Time_Graph_after3.png
          76 kB
        17. Response_Time_Graph_before1.png
          Response_Time_Graph_before1.png
          55 kB
        18. Response_Time_Graph_before2.png
          Response_Time_Graph_before2.png
          55 kB
        19. Response_Time_Graph_before3.png
          Response_Time_Graph_before3.png
          82 kB
        20. Response_Time_over_time_after1.png
          Response_Time_over_time_after1.png
          134 kB
        21. Response_Time_over_time_after2.png
          Response_Time_over_time_after2.png
          130 kB
        22. Response_Time_over_time_after3.png
          Response_Time_over_time_after3.png
          101 kB
        23. Response_Time_over_time_before1.png
          Response_Time_over_time_before1.png
          58 kB
        24. Response_Time_over_time_before2.png
          Response_Time_over_time_before2.png
          82 kB
        25. Response_Time_over_time_before3.png
          Response_Time_over_time_before3.png
          89 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  4 years, 15 weeks, 6 days ago

                  Packages

                  Version Package
                  6.2.4 CE GA5
                  6.2.X EE