Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-54348

Add the ability to disable module framework

    Details

      Description

      Description

      Module framework related thread Refresh Packages may cause SIGBUS error intermittently and makes portal's JVM to crash on Linux and Solaris.

      ---------------  T H R E A D  ---------------
      
      Current thread (0x00007fdee401f800):  JavaThread "Refresh Packages" daemon [_thread_in_native, id=8885, stack(0x00007fdf258be000,0x00007fdf259bf000)]
      
      siginfo:si_signo=SIGBUS: si_errno=0, si_code=2 (BUS_ADRERR), si_addr=0x00007fdf25ad4d41
      
      Registers:
      RAX=0x00007fdef00e16a0, RBX=0x00007fdef4c933d0, RCX=0x00007fdef00e1690, RDX=0x00007fdef0000040
      RSP=0x00007fdf259bac00, RBP=0x00007fdf259bac50, RSI=0x00007fdef0000038, RDI=0x00007fdef03ce380
      R8 =0x0000000000000003, R9 =0x00000000000000ff, R10=0x0000000000000048, R11=0x00007fdf9d081bf8
      R12=0x00007fdf25ad4d24, R13=0x00007fdef00e16a0, R14=0x0000000011f5ecc2, R15=0x00007fdef49bc6a0
      RIP=0x00007fdfa51dd982, EFLAGS=0x0000000000010206, CSGSFS=0x9f81000000000033, ERR=0x0000000000000004
        TRAPNO=0x000000000000000e
      
      Top of Stack: (sp=0x00007fdf259bac00)
      0x00007fdf259bac00:   00000007a0000c90 00007fde00000000
      0x00007fdf259bac10:   00007fdef49bc7c0 00007fdfa71f22ec
      0x00007fdf259bac20:   00007fdef00e16a0 00007fdef4c933d0
      0x00007fdf259bac30:   00007fdef49bc6a0 00007fdee400d830
      0x00007fdf259bac40:   0000000011f5ecc2 00007fdf259bacb0
      0x00007fdf259bac50:   00007fdf259bac90 00007fdfa51de0b0
      0x00007fdf259bac60:   0000003d259bac90 00007fdf259bacb0
      0x00007fdf259bac70:   000000000000003d 00007fdee401f9e8
      0x00007fdf259bac80:   00007fdf259bacb0 00007fdf259bb108
      0x00007fdf259bac90:   00007fdf259bb0f0 00007fdfa51dceed
      0x00007fdf259baca0:   01c6e23b3749b284 00007fdef49bc6a0
      0x00007fdf259bacb0:   696c63652f67726f 696d65672f657370
      0x00007fdf259bacc0:   2f746d676d2f696e 6f74617669746341
      0x00007fdf259bacd0:   536e6165424d2472 7254656369767265
      0x00007fdf259bace0:   2e322472656b6361 0000007373616c63
      0x00007fdf259bacf0:   0000000000000000 00007fdf259bada0
      0x00007fdf259bad00:   00007fdee401f800 00007fdf28474828
      0x00007fdf259bad10:   0000000000000001 0000000000000000
      0x00007fdf259bad20:   00007fdf259bad50 00007fdee401f800
      0x00007fdf259bad30:   00007fdf259bada0 00007fdf28474828
      0x00007fdf259bad40:   0000000000000000 00007fdee401f800
      0x00007fdf259bad50:   00007fdf259bae00 00007fdfa71e3d08
      0x00007fdf259bad60:   00007fdf259bad88 00007fdf2870bd40
      0x00007fdf259bad70:   00007fdf2870bd50 00007fdf2870c128
      0x00007fdf259bad80:   00007fdee401d1f0 00000000000003d8
      0x00007fdf259bad90:   0000000000000000 0000000000000000
      0x00007fdf259bada0:   00007fdfa00089a0 00007fdee401f800
      0x00007fdf259badb0:   00007fdf259bade0 00007fdfa6fa2660
      0x00007fdf259badc0:   00007fdf259bade0 00007fdfa7858e50
      0x00007fdf259badd0:   00007fdfa00089a0 0000000000000004
      0x00007fdf259bade0:   00007fdf259bae00 00007fdfa6e3434a
      0x00007fdf259badf0:   00000007a0179fb0 0000000000000da1 
      
      Instructions: (pc=0x00007fdfa51dd982)
      0x00007fdfa51dd962:   00 48 c7 40 28 00 00 00 00 41 80 7f 30 00 0f 84
      0x00007fdfa51dd972:   8a 02 00 00 4c 8b 63 08 4d 2b 67 28 4d 03 67 18
      0x00007fdfa51dd982:   41 0f b6 5c 24 1d 41 0f b6 44 24 1c c1 e3 08 09
      0x00007fdfa51dd992:   c3 41 0f b6 44 24 1e 88 45 bd 41 0f b6 54 24 20 
      
      Register to memory mapping:
      
      RAX=0x00007fdef00e16a0 is an unknown value
      RBX=0x00007fdef4c933d0 is an unknown value
      RCX=0x00007fdef00e1690 is an unknown value
      RDX=0x00007fdef0000040 is an unknown value
      RSP=0x00007fdf259bac00 is pointing into the stack for thread: 0x00007fdee401f800
      RBP=0x00007fdf259bac50 is pointing into the stack for thread: 0x00007fdee401f800
      RSI=0x00007fdef0000038 is an unknown value
      RDI=0x00007fdef03ce380 is an unknown value
      R8 =0x0000000000000003 is an unknown value
      R9 =0x00000000000000ff is an unknown value
      R10=0x0000000000000048 is an unknown value
      R11=0x00007fdf9d081bf8 is at entry_point+24 in (nmethod*)0x00007fdf9d081a50
      R12=0x00007fdf25ad4d24 is an unknown value
      R13=0x00007fdef00e16a0 is an unknown value
      R14=0x0000000011f5ecc2 is an unknown value
      R15=0x00007fdef49bc6a0 is an unknown value
      
      
      Stack: [0x00007fdf258be000,0x00007fdf259bf000],  sp=0x00007fdf259bac00,  free space=1011k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C  [libzip.so+0x4982]  newEntry+0x62
      C  [libzip.so+0x50b0]  ZIP_GetEntry+0xd0
      C  [libzip.so+0x3eed]  Java_java_util_zip_ZipFile_getEntry+0xad
      J 151  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007fdf9d081cae [0x00007fdf9d081be0+0xce]
      j  java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+38
      j  org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getZipEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+28
      j  org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getEntry(Ljava/lang/String;)Lorg/eclipse/osgi/baseadaptor/bundlefile/BundleEntry;+11
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(Ljava/lang/String;Lorg/eclipse/osgi/baseadaptor/loader/ClasspathEntry;[Lorg/eclipse/osgi/baseadaptor/hooks/ClassLoadingStatsHook;)Ljava/lang/Class;+67
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(Ljava/lang/String;[Lorg/eclipse/osgi/baseadaptor/hooks/ClassLoadingStatsHook;)Ljava/lang/Class;+28
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+90
      j  org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+5
      j  org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+47
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+183
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClass(Ljava/lang/String;Z)Ljava/lang/Class;+32
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+3
      j  org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+82
      j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+40
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6000f5]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
      V  [libjvm.so+0x5feb58]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
      V  [libjvm.so+0x5fee27]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
      V  [libjvm.so+0x5feef6]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Handle, Thread*)+0x56
      V  [libjvm.so+0x923539]  SystemDictionary::load_instance_class(Symbol*, Handle, Thread*)+0x419
      V  [libjvm.so+0x923ce1]  SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*)+0x791
      V  [libjvm.so+0x924144]  SystemDictionary::resolve_or_null(Symbol*, Handle, Handle, Thread*)+0x54
      V  [libjvm.so+0x925224]  SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*)+0x34
      V  [libjvm.so+0x490f48]  constantPoolOopDesc::klass_at_impl(constantPoolHandle, int, Thread*)+0x278
      V  [libjvm.so+0x5f8084]  InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int)+0x74
      j  org.eclipse.gemini.mgmt.Activator$MBeanServiceTracker.removedService(Lorg/osgi/framework/ServiceReference;Ljava/lang/Object;)V+53
      j  org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(Lorg/osgi/framework/ServiceReference;Lorg/osgi/framework/ServiceEvent;Ljava/lang/Object;)V+9
      j  org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V+13
      j  org.osgi.util.tracker.AbstractTracked.untrack(Ljava/lang/Object;Ljava/lang/Object;)V+74
      j  org.osgi.util.tracker.ServiceTracker.close()V+86
      j  org.eclipse.gemini.mgmt.Activator.stop(Lorg/osgi/framework/BundleContext;)V+4
      j  org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run()Ljava/lang/Object;+29
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6000f5]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
      V  [libjvm.so+0x5feb58]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
      V  [libjvm.so+0x6857f4]  JVM_DoPrivileged+0x2a4
      j  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;)Ljava/lang/Object;+0
      j  org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop()V+8
      j  org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(I)V+129
      j  org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(Z)V+15
      j  org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;Z)Z+46
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.suspendBundle(Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;)V+20
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.processDelta([Lorg/eclipse/osgi/service/resolver/BundleDelta;ZLorg/eclipse/osgi/service/resolver/State;)[Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;+423
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.doResolveBundles([Lorg/osgi/framework/Bundle;Z[Lorg/osgi/framework/FrameworkListener;)V+471
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl$1.run()V+13
      j  java.lang.Thread.run()V+11
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6000f5]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
      V  [libjvm.so+0x5feb58]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
      V  [libjvm.so+0x5fee27]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197
      V  [libjvm.so+0x5fef47]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
      V  [libjvm.so+0x67bf65]  thread_entry(JavaThread*, Thread*)+0xe5
      V  [libjvm.so+0x95a08f]  JavaThread::thread_main_inner()+0xdf
      V  [libjvm.so+0x95a195]  JavaThread::run()+0xf5
      V  [libjvm.so+0x820c28]  java_start(Thread*)+0x108
      
      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      J 151  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007fdf9d081c38 [0x00007fdf9d081be0+0x58]
      j  java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+38
      j  org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getZipEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+28
      j  org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getEntry(Ljava/lang/String;)Lorg/eclipse/osgi/baseadaptor/bundlefile/BundleEntry;+11
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(Ljava/lang/String;Lorg/eclipse/osgi/baseadaptor/loader/ClasspathEntry;[Lorg/eclipse/osgi/baseadaptor/hooks/ClassLoadingStatsHook;)Ljava/lang/Class;+67
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(Ljava/lang/String;[Lorg/eclipse/osgi/baseadaptor/hooks/ClassLoadingStatsHook;)Ljava/lang/Class;+28
      j  org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+90
      j  org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+5
      j  org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+47
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+183
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClass(Ljava/lang/String;Z)Ljava/lang/Class;+32
      j  org.eclipse.osgi.internal.loader.BundleLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+3
      j  org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+82
      j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+40
      v  ~StubRoutines::call_stub
      j  org.eclipse.gemini.mgmt.Activator$MBeanServiceTracker.removedService(Lorg/osgi/framework/ServiceReference;Ljava/lang/Object;)V+53
      j  org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(Lorg/osgi/framework/ServiceReference;Lorg/osgi/framework/ServiceEvent;Ljava/lang/Object;)V+9
      j  org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V+13
      j  org.osgi.util.tracker.AbstractTracked.untrack(Ljava/lang/Object;Ljava/lang/Object;)V+74
      j  org.osgi.util.tracker.ServiceTracker.close()V+86
      j  org.eclipse.gemini.mgmt.Activator.stop(Lorg/osgi/framework/BundleContext;)V+4
      j  org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run()Ljava/lang/Object;+29
      v  ~StubRoutines::call_stub
      j  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;)Ljava/lang/Object;+0
      j  org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop()V+8
      j  org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(I)V+129
      j  org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(Z)V+15
      j  org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;Z)Z+46
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.suspendBundle(Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;)V+20
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.processDelta([Lorg/eclipse/osgi/service/resolver/BundleDelta;ZLorg/eclipse/osgi/service/resolver/State;)[Lorg/eclipse/osgi/framework/internal/core/AbstractBundle;+423
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl.doResolveBundles([Lorg/osgi/framework/Bundle;Z[Lorg/osgi/framework/FrameworkListener;)V+471
      j  org.eclipse.osgi.framework.internal.core.PackageAdminImpl$1.run()V+13
      j  java.lang.Thread.run()V+11
      v  ~StubRoutines::call_stub
      

      Analyses

      JDK 6u21 release includes a new system property, sun.zip.disableMemoryMapping, which allows the user to disable the mmap usage in Sun's java.util.zip.Zipfile implementation (on Solaris and Linux platforms). Solaris or Linux applications that use java.util.zip.ZipFile may experience a SIGBUS VM crash if the application accidentally overwrites any zip or jar files that are still being used by the same Java runtime. Although this is a programming error of the offending application, this system property provides a solution to avoid the VM crash. With the property set to true (-Dsun.zip.disableMemoryMapping=true, or simply -Dsun.zip.disableMemoryMapping) the Sun JDK/JRE runtime disables the mmap usage and the VM crash that might otherwise occur by overwriting the jar or zip file can be avoided.

      As the release notes of JDK states, this might be a bug in the OSGi framework, thus we need to be able to disable the module framework on ee-6.2.x.

      Steps to reproduce

      This is an intermittent issue, no steps can be given.

      Workaround

      Apply system property -Dsun.zip.disableMemoryMapping=true to the JVM's argument list.

      Notes for QA

      Although steps to reproduce the SIGBUS issue haven't been found yet, it can be tested if the fix disables the module framework indeed.

      1) Start Liferay
      2) Take a thread dump

      Checkpoint: Observe that there are module framework related threads.

      "Framework Event Dispatcher" daemon prio=10 tid=0x00007f20584fd800 nid=0x7693 in Object.wait() [0x00007f20a416d000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x0000000700b8cb80> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
              at java.lang.Object.wait(Object.java:485)
              at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
              - locked <0x0000000700b8cb80> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
              at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
      
         Locked ownable synchronizers:
              - None
      
      "Framework Active Thread" prio=10 tid=0x00007f2058036000 nid=0x7692 in Object.wait() [0x00007f20a4286000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              - waiting on <0x00000007007ea858> (a org.eclipse.osgi.framework.internal.core.Framework)
              at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1863)
              - locked <0x00000007007ea858> (a org.eclipse.osgi.framework.internal.core.Framework)
              at java.lang.Thread.run(Thread.java:662)
      
         Locked ownable synchronizers:
              - None
      

      3) Set module.framework.enabled=false and restart Liferay

      Checkpoint: The aforementioned thread should disappear.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hong.zhao Hong Zhao
                Reporter:
                laszlo.csontos Laszlo Csontos (Inactive)
                Participants of an Issue:
                Recent user:
                Esther Sanz
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  4 years, 32 weeks ago

                  Packages

                  Version Package
                  6.2.3 CE GA4
                  6.2.X EE