Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-54888

Staging publish fails with error indicating Requested array size exceeds VM limit

    Details

      Description

      Short description of the client's issue:
      The client is attempting to perform a publish test after adding several files to the environment. They are now receiving an error, and all subsequent publish attempts are running into the following error:

      The publication process did not start due to validation errors.
      Requested array size exceeds VM limit

      08:05:54,689 INFO  [liferay/scheduler_dispatch-1][PluginPackageUtil:1280] Reloading repositories
      11:59:43,685 ERROR [http-/0.0.0.0:8443-7][TunnelServlet:86] java.lang.reflect.InvocationTargetException
      java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.liferay.portal.kernel.util.MethodHandler.invoke(MethodHandler.java:61)
      	at com.liferay.portal.servlet.TunnelServlet.doPost(TunnelServlet.java:79)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:169)
      	at com.liferay.portal.servlet.filters.authverifier.AuthVerifierFilter.processFilter(AuthVerifierFilter.java:169)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:59)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:169)
      	at com.liferay.portal.servlet.filters.jsoncontenttype.JSONContentTypeFilter.processFilter(JSONContentTypeFilter.java:42)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:59)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:169)
      	at com.liferay.portal.servlet.filters.sso.ntlm.NtlmPostFilter.processFilter(NtlmPostFilter.java:83)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:59)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
      	at com.liferay.portal.kernel.servlet.BaseFilter.processFilter(BaseFilter.java:169)
      	at com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.processFilter(VirtualHostFilter.java:226)
      	at com.liferay.portal.kernel.servlet.BaseFilter.doFilter(BaseFilter.java:59)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:185)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
      	at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:738)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:185)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
      	at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:96)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:407)
      	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
      	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
      	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
      	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
      	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
      	at java.util.Arrays.copyOf(Arrays.java:2271)
      	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
      	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
      	at org.hibernate.type.descriptor.java.DataHelper.extractBytes(DataHelper.java:170)
      	at org.hibernate.type.descriptor.java.BlobTypeDescriptor.unwrap(BlobTypeDescriptor.java:123)
      	at org.hibernate.type.descriptor.java.BlobTypeDescriptor.unwrap(BlobTypeDescriptor.java:47)
      	at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$4$1.doBind(BlobTypeDescriptor.java:101)
      	at org.hibernate.type.descriptor.sql.BasicBinder.bind(BasicBinder.java:91)
      	at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:283)
      	at org.hibernate.type.AbstractStandardBasicType.nullSafeSet(AbstractStandardBasicType.java:278)
      	at org.hibernate.type.AbstractSingleColumnStandardBasicType.nullSafeSet(AbstractSingleColumnStandardBasicType.java:89)
      	at org.hibernate.persister.entity.AbstractEntityPersister.dehydrate(AbstractEntityPersister.java:2184)
      	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2559)
      	at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2495)
      	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2822)
      	at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:113)
      	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:273)
      	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:265)
      	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
      	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
      	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)
      	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
      	at com.liferay.portal.dao.orm.hibernate.SessionImpl.flush(SessionImpl.java:144)
      	at com.liferay.portal.kernel.dao.orm.ClassLoaderSession.flush(ClassLoaderSession.java:237)
      	at com.liferay.portlet.documentlibrary.service.persistence.DLContentPersistenceImpl.updateImpl(DLContentPersistenceImpl.java:2455)
      	at com.liferay.portlet.documentlibrary.service.persistence.DLContentPersistenceImpl.updateImpl(DLContentPersistenceImpl.java:1)
      	at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.update(BasePersistenceImpl.java:333)
      	at com.liferay.portlet.documentlibrary.service.impl.DLContentLocalServiceImpl.addContent(DLContentLocalServiceImpl.java:91)
      	at sun.reflect.GeneratedMethodAccessor1216.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)

      Basically, it looks like the LAR file is pushed over to the live environment, attempts to perform the extraction/import, and encounters this OutOfMemory error.

      The client has indicated that their current JVM settings (in regards to memory are the exact same in both of their environments.

      JAVA_OPTS

      OD Server (aka LIVE):

      set "JAVA_OPTS=-Xrs"
      
      set "JAVA_OPTS=%JAVA_OPTS% -server -d64"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSClassUnloadingEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSPermGenSweepingEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:NewSize=4096m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxNewSize=4096m"
      set "JAVA_OPTS=%JAVA_OPTS% -Xms8192m"
      set "JAVA_OPTS=%JAVA_OPTS% -Xmx8192m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:PermSize=512m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:SurvivorRatio=65536"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:TargetSurvivorRatio=0"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxTenuringThreshold=0"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseParNewGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:ParallelGCThreads=16"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSParallelRemarkEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSCompactWhenClearAllSoftRefs"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:CMSInitiatingOccupancyFraction=85"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSScavengeBeforeRemark"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSConcurrentMTEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:ParallelCMSThreads=2"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseLargePages"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:LargePageSizeInBytes=256m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseCompressedOops"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+DisableExplicitGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:-UseBiasedLocking"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+BindGCTaskThreadsToCPUs"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseFastAccessorMethods"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+HeapDumpOnOutOfMemoryError"
      
      rem # Prefer IPv4
      set "JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true"
      
      rem # Make Byteman classes visible in all module loaders
      rem # This is necessary to inject Byteman rules into AS7 deployments
      set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman"
      
      rem # Sample JPDA settings for remote socket debugging
      rem set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n"
      
      rem # Sample JPDA settings for shared memory debugging
      rem set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_shmem,address=jboss,server=y,suspend=n"
      
      rem # Use JBoss Modules lockless mode
      rem set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.lockless=true"
      
      set "JAVA_OPTS=%JAVA_OPTS% -Dfile.encoding=UTF-8 -Duser.timezone=EST"

      STAGE Server:

      set "JAVA_OPTS=-Xrs"
      
      set "JAVA_OPTS=%JAVA_OPTS% -server -d64"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSClassUnloadingEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSPermGenSweepingEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:NewSize=4096m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxNewSize=4096m"
      set "JAVA_OPTS=%JAVA_OPTS% -Xms8192m"
      set "JAVA_OPTS=%JAVA_OPTS% -Xmx8192m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:PermSize=512m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512m"
      rem Try to keep Suvivor space half full. I think 65k is too high.
      rem set "JAVA_OPTS=%JAVA_OPTS% -XX:SurvivorRatio=65536"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:SurvivorRatio=500"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:TargetSurvivorRatio=0"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:MaxTenuringThreshold=0"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseParNewGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:ParallelGCThreads=16"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSParallelRemarkEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSCompactWhenClearAllSoftRefs"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:CMSInitiatingOccupancyFraction=85"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSScavengeBeforeRemark"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+CMSConcurrentMTEnabled"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:ParallelCMSThreads=2"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseLargePages"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:LargePageSizeInBytes=256m"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseCompressedOops"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+DisableExplicitGC"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:-UseBiasedLocking"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+BindGCTaskThreadsToCPUs"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+UseFastAccessorMethods"
      rem # Add GC logging - rchu 20140906
      set "JAVA_OPTS=%JAVA_OPTS% -Xloggc:E:\Liferay\logs\liferaygc1.log"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+PrintGCDetails"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+PrintGCApplicationConcurrentTime"
      set "JAVA_OPTS=%JAVA_OPTS% -XX:+PrintGCApplicationStoppedTime"
      
      rem # Prefer IPv4
      set "JAVA_OPTS=%JAVA_OPTS% -Djava.net.preferIPv4Stack=true"
      
      rem # Make Byteman classes visible in all module loaders
      rem # This is necessary to inject Byteman rules into AS7 deployments
      set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman"
      
      rem # Sample JPDA settings for remote socket debugging
      rem set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n"
      
      rem # Sample JPDA settings for shared memory debugging
      rem set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_shmem,address=jboss,server=y,suspend=n"
      
      rem # Use JBoss Modules lockless mode
      rem set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.lockless=true"
      
      rem set "JAVA_OPTS=%JAVA_OPTS% -Dfile.encoding=UTF-8 -Duser.timezone=EST"
      set "JAVA_OPTS=%JAVA_OPTS% -Dfile.encoding=UTF-8 -Duser.timezone=America/New_York"

      As you can see, the relevant properties here are that the heap is set to 8192m in both environments.

      Steps to reproduce / testing done:
      I have not been able to reproduce the OutOfMemory issue at hand as I am able to recreate the conditions that are required to establish the root cause.

      Results of steps / testing:
      N/A

      Errors/logs:
      FROM THE OD (LIVE) SERVER:
      liferay-OD.2015-03-31.log
      liferay-OD.2015-04-01.log

      FROM THE STAGE SERVER:
      liferay-stage.2015-03-31.log
      liferay-stage.2015-04-01.log

      Notice that at the end of STAGE's logs, we can see that the file exports fine, and is pushed over to OD. It seems like the issue is occurring on OD's side, but this is inconclusive. It looks like there may be something wrong with the memory settings considering that we're getting an OutOfMemoryError.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              christopher.kian Christopher Kian
              Reporter:
              christopher.kian Christopher Kian
              Participants of an Issue:
              Recent user:
              Tibor Jandi
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                4 years, 12 weeks, 6 days ago

                  Packages

                  Version Package