Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-54672

Writing to search index destroys current transaction meta data when Index Writer queue is full (ArrayIndexOutOfBoundsException: -1)

    Details

      Description

      Steps to reproduce:

      1. Install Liferay 6.1.30 EE GA3 with the latest patches (currently portal-35-6130.zip)
      2. Create a portal-ext.properties file with this line added:
      index.search.writer.max.queue.size = 1
      3. Start the portal
      4. Deploy one of the attached portlets:

      • OR you can use the portlet: create300users.war that needs to be deployed as a portlet and it can be found under the "sample" portlet category. You can just add the portlet to the main page and click on the ActionURL to start it.

      Expected Results:
      300 users should be created without any errors

      Actual Results:
      User creation starts, but an error occurs:

      java.lang.ArrayIndexOutOfBoundsException: -1
              at java.util.ArrayList.elementData(ArrayList.java:400)
              at java.util.ArrayList.remove(ArrayList.java:477)
              at com.liferay.portal.spring.transaction.TransactionCommitCallbackUtil.popCallbackList(TransactionCommitCallbackUtil.java:64)
              at com.liferay.portal.spring.transaction.BaseTransactionExecutor.invokeCallbacks(BaseTransactionExecutor.java:31)
              at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.processCommit(DefaultTransactionExecutor.java:120)
              at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.execute(DefaultTransactionExecutor.java:70)
              at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:52)
              at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:113)
              at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
              at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:113)
              at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)

      Reproduced with the customer's patches (since liferay-hotfix-1505-6130.zip)
      Reproduced with the latest patches for 6.1 EE GA3
      Reproduced on 6.1.x (ca05ec51cf697019c59445c396bf5bac58364164)
      Reproduced on 6.2.x (392e4d66a78316ca4dd1935192e68b9596aae90f)
      Could not reproduce on Master, but the related code didn't change, so if I could force a CallerRunsPolicy to run, the same thing should happen.

      Additional info:
      "AbstractSearchEngineConfigurator registers a RejectedExecutionHandler to the search writer destination when the property index.search.writer.max.queue.size is set (which is the case since liferay-hotfix-1505-6130) so that, when the queue is full, indexing is done on the current thread instead of an executor thread in background.
      Unfortunately, the Runnables that ParallelDestination produces are made for running on a exceutor thread and call CentralizedThreadLocal.clearShortLivedThreadLocals(); in a finally block. This deletes the callback lists TransactionCommitCallbackUtil manages.
      On the next transaction boundary (when TransactionCommitCallbackUtil.popCallbackList() is called) the transaction handling explodes:
      java.lang.ArrayIndexOutOfBoundsException: -1
      "

      Solution:
      The solution I could come up with is not to mace TransactionCallbackThreadLocal an AutoResetThreadLocal, so it won't be cleared by the CentralizedThreadLocal.clearShortLivedThreadLocals() call.

      WORKAROUND
      As a workaround of this LPS, it is possible to increment the max queue size to avoid the execution of the problematic code:

      index.search.writer.max.queue.size=9999999
      

      It is also good to avoid the creation of threads batching lucene writes, this makes disk writes less often:

      lucene.commit.batch.size=10000
      lucene.commit.time.interval=300000
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  4 years, 19 weeks, 1 day ago

                  Packages

                  Version Package
                  6.1.X EE
                  6.2.4 CE GA5
                  6.2.X EE
                  7.0.0 M5