OutOfMemory risk in *Indexer with millions of records

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 6.2.0 CE B1
• Fix Version/s: 6.1.X EE, 6.2.0 CE RC3, 6.2.0 CE RC5
• Component/s: Core Infrastructure, Performance
• Labels:
  • 6.1_release_notes
  • liferay-fixpack-administration-4-6130
  • liferay-fixpack-collaboration-8-6130
  • liferay-fixpack-document-management-11-6130
  • liferay-fixpack-misc-3-6130
  • liferay-fixpack-platform-19-6130
  • liferay-fixpack-web-content-10-6130
• Branch Version/s:
  6.1.x
• Backported to Branch:
  Committed
• Fix Priority: 4
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/14368,https://github.com/brianchandotcom/liferay-plugins/pull/2145

I have an installation with 7 companies and one of them has 1,3 millions of users.

When I tried to do an users reindex the system goes OutOf Memory.
My JVM is configured with "-Xmx1024m". I know it is few memory but IMHO the reindex process will have to load a bunch of records, update the SearchEngine, and discard it.

During my analisys I notice that UserIndexer use an ActionableDynamicQuery to fill up an array of DocumentImpl. After that all the records of the selected company are processed the array is submitted to IndexerWriter.

I've used an ext (indexer-ext-6.2.0.1.war) to help me:

• speedup the processing of companies with few users (look UserActionableDynamicQueryExt)
• increase logging

The files "catalina-02.out", "heap-histo-02.log" and JVisualVM snapshot "application-1376574827825.apps" shows how heap usage increases constantly for the company with millions of users.

In 6.1 strategy was to invoke the IndexWriter for every bunch of records.

I've extended my ext to call IndexWriter every 5000 DocumentImpl.
The files "catalina-03.out" and "heap-histo-03.log" show how heap usage is lower.