Details

    • Branch Version/s:
      6.1.x
    • Backported to Branch:
      Committed
    • Fix Priority:
      4

      Description

      I have an installation with 7 companies and one of them has 1,3 millions of users.

      When I tried to do an users reindex the system goes OutOf Memory.
      My JVM is configured with "-Xmx1024m". I know it is few memory but IMHO the reindex process will have to load a bunch of records, update the SearchEngine, and discard it.

      During my analisys I notice that UserIndexer use an ActionableDynamicQuery to fill up an array of DocumentImpl. After that all the records of the selected company are processed the array is submitted to IndexerWriter.

      I've used an ext (indexer-ext-6.2.0.1.war) to help me:

      • speedup the processing of companies with few users (look UserActionableDynamicQueryExt)
      • increase logging

      The files "catalina-02.out", "heap-histo-02.log" and JVisualVM snapshot "application-1376574827825.apps" shows how heap usage increases constantly for the company with millions of users.

      In 6.1 strategy was to invoke the IndexWriter for every bunch of records.

      I've extended my ext to call IndexWriter every 5000 DocumentImpl.
      The files "catalina-03.out" and "heap-histo-03.log" show how heap usage is lower.

        Attachments

        1. application-1376574827825.apps
          39 kB
        2. catalina-02.out
          129 kB
        3. catalina-03.out
          141 kB
        4. heap-histo-02.log
          226 kB
        5. heap-histo-03.log
          748 kB
        6. indexer-ext-6.2.0.1.zip
          69 kB
        7. indexer-ext-6.2.0.2.war
          70 kB
        8. Screen Shot 2013-10-22 at 5.27.42 PM.png
          Screen Shot 2013-10-22 at 5.27.42 PM.png
          309 kB
        9. Screen Shot 2013-10-22 at 5.27.54 PM.png
          Screen Shot 2013-10-22 at 5.27.54 PM.png
          366 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  3 years, 39 weeks, 2 days ago