Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-39087

OutOfMemory risk in *Indexer with millions of records

    Description

    I have an installation with 7 companies and one of them has 1,3 millions of users.

    When I tried to do an users reindex the system goes OutOf Memory.
    My JVM is configured with "-Xmx1024m". I know it is few memory but IMHO the reindex process will have to load a bunch of records, update the SearchEngine, and discard it.

    During my analisys I notice that UserIndexer use an ActionableDynamicQuery to fill up an array of DocumentImpl. After that all the records of the selected company are processed the array is submitted to IndexerWriter.

    I've used an ext (indexer-ext-6.2.0.1.war) to help me:

    • speedup the processing of companies with few users (look UserActionableDynamicQueryExt)
    • increase logging

    The files "catalina-02.out", "heap-histo-02.log" and JVisualVM snapshot "application-1376574827825.apps" shows how heap usage increases constantly for the company with millions of users.

    In 6.1 strategy was to invoke the IndexWriter for every bunch of records.

    I've extended my ext to call IndexWriter every 5000 DocumentImpl.
    The files "catalina-03.out" and "heap-histo-03.log" show how heap usage is lower.

      Attachments

      1. application-1376574827825.apps
        39 kB
        Mauro Mariuzzo
      2. catalina-02.out
        129 kB
        Mauro Mariuzzo
      3. catalina-03.out
        141 kB
        Mauro Mariuzzo
      4. heap-histo-02.log
        226 kB
        Mauro Mariuzzo
      5. heap-histo-03.log
        748 kB
        Mauro Mariuzzo
      6. indexer-ext-6.2.0.2.war
        70 kB
        Mauro Mariuzzo
      7. Screen Shot 2013-10-22 at 5.27.42 PM.png
        309 kB
        James Falkner
      8. Screen Shot 2013-10-22 at 5.27.54 PM.png
        366 kB
        James Falkner

        Issue Links

          Activity

            People

            Assignee:
            michael.han Michael Han (Inactive)
            Reporter:
            maumar Mauro Mariuzzo
            Participants of an Issue:
            Recent user:
            Esther Sanz
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since last comment:
              8 years, 6 weeks, 3 days ago

                Packages

                Version Package
                6.1.X EE
                6.2.0 CE RC3
                6.2.0 CE RC5