Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-39087

OutOfMemory risk in *Indexer with millions of records

    Description

    I have an installation with 7 companies and one of them has 1,3 millions of users.

    When I tried to do an users reindex the system goes OutOf Memory.
    My JVM is configured with "-Xmx1024m". I know it is few memory but IMHO the reindex process will have to load a bunch of records, update the SearchEngine, and discard it.

    During my analisys I notice that UserIndexer use an ActionableDynamicQuery to fill up an array of DocumentImpl. After that all the records of the selected company are processed the array is submitted to IndexerWriter.

    I've used an ext (indexer-ext-6.2.0.1.war) to help me:

    • speedup the processing of companies with few users (look UserActionableDynamicQueryExt)
    • increase logging

    The files "catalina-02.out", "heap-histo-02.log" and JVisualVM snapshot "application-1376574827825.apps" shows how heap usage increases constantly for the company with millions of users.

    In 6.1 strategy was to invoke the IndexWriter for every bunch of records.

    I've extended my ext to call IndexWriter every 5000 DocumentImpl.
    The files "catalina-03.out" and "heap-histo-03.log" show how heap usage is lower.

      Attachments

      1. application-1376574827825.apps
        39 kB
      2. catalina-02.out
        129 kB
      3. catalina-03.out
        141 kB
      4. heap-histo-02.log
        226 kB
      5. heap-histo-03.log
        748 kB
      6. indexer-ext-6.2.0.1.zip
        69 kB
      7. indexer-ext-6.2.0.2.war
        70 kB
      8. Screen Shot 2013-10-22 at 5.27.42 PM.png
        Screen Shot 2013-10-22 at 5.27.42 PM.png
        309 kB
      9. Screen Shot 2013-10-22 at 5.27.54 PM.png
        Screen Shot 2013-10-22 at 5.27.54 PM.png
        366 kB

        Issue Links

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                6 years, 48 weeks, 3 days ago

                Packages

                Version Package
                6.1.X EE
                6.2.0 CE RC3
                6.2.0 CE RC5