Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-39087

OutOfMemory risk in *Indexer with millions of records


    I have an installation with 7 companies and one of them has 1,3 millions of users.

    When I tried to do an users reindex the system goes OutOf Memory.
    My JVM is configured with "-Xmx1024m". I know it is few memory but IMHO the reindex process will have to load a bunch of records, update the SearchEngine, and discard it.

    During my analisys I notice that UserIndexer use an ActionableDynamicQuery to fill up an array of DocumentImpl. After that all the records of the selected company are processed the array is submitted to IndexerWriter.

    I've used an ext (indexer-ext- to help me:

    • speedup the processing of companies with few users (look UserActionableDynamicQueryExt)
    • increase logging

    The files "catalina-02.out", "heap-histo-02.log" and JVisualVM snapshot "application-1376574827825.apps" shows how heap usage increases constantly for the company with millions of users.

    In 6.1 strategy was to invoke the IndexWriter for every bunch of records.

    I've extended my ext to call IndexWriter every 5000 DocumentImpl.
    The files "catalina-03.out" and "heap-histo-03.log" show how heap usage is lower.


      1. application-1376574827825.apps
        39 kB
      2. catalina-02.out
        129 kB
      3. catalina-03.out
        141 kB
      4. heap-histo-02.log
        226 kB
      5. heap-histo-03.log
        748 kB
      6. indexer-ext-
        69 kB
      7. indexer-ext-
        70 kB
      8. Screen Shot 2013-10-22 at 5.27.42 PM.png
        Screen Shot 2013-10-22 at 5.27.42 PM.png
        309 kB
      9. Screen Shot 2013-10-22 at 5.27.54 PM.png
        Screen Shot 2013-10-22 at 5.27.54 PM.png
        366 kB

      Issue Links



            michael.han Michael Han (Inactive)
            maumar Mauro Mariuzzo
            Rafaela Nascimento Rafaela Nascimento
            0 Vote for this issue
            6 Start watching this issue


              9 years, 33 weeks, 3 days ago


                Version Package
                6.1.X EE
                6.2.0 CE RC3
                6.2.0 CE RC5