Tomcat 8 Shutdown error - IllegalStateException logging info, the web application instance has been stopped already

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.0 Beta 2
• Fix Version/s: 6.1.X EE, 6.2.X EE, 7.0.0 Beta 5
• Component/s: Environments, Environments > App Servers
• Labels:
  • QA-AUTOMATION
  • liferay-fixpack-de-23-7010
  • liferay-fixpack-portal-110-6210
• Branch Version/s:
  6.2.x, 6.1.x
• Backported to Branch:
  Committed
• Fix Priority: 5
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/35091

This error happens on shutdown, and if the user uses the shutdown.bat/sh command (which is also what our CI uses), then the tomcat JVM hangs.

This issue also is not OS-specific and can be reproduced in linux and windows (tested using Tomcat 8.0.30).

Steps to reproduce:

1. Start Liferay
2. Shut down Liferay using shutdown.bat / shutdown.sh (Note that if you use ctrl + c, you will get the illegalStateException but the shutdown process will not hang).
   • The tomcat JVM will hang.
   • Immediately on shutdown, an illegalStateException appears:

     13-Jan-2016 03:01:53.178 INFO [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.springframework.web.context.ContextCleanupListener]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
      java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.springframework.web.context.ContextCleanupListener]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
             at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1353)
             at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1341)
             at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
             at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
             at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:145)
             at com.liferay.portal.spring.context.PortalContextLoaderListener.contextDestroyed(PortalContextLoaderListener.java:168)
             at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4859)
             at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5478)
             at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
             at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
             at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
             at java.lang.Thread.run(Thread.java:745)
     
     13-Jan-2016 03:01:53.181 SEVERE [localhost-startStop-2] org.apache.catalina.core.StandardContext.listenerStop Exception sending context destroyed event to listener instance of class com.liferay.portal.spring.context.PortalContextLoaderListener
      java.lang.NoClassDefFoundError: org/springframework/web/context/ContextCleanupListener
             at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:145)
             at com.liferay.portal.spring.context.PortalContextLoaderListener.contextDestroyed(PortalContextLoaderListener.java:168)
             at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4859)
             at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5478)
             at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
             at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
             at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
             at java.lang.Thread.run(Thread.java:745)
     Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.springframework.web.context.ContextCleanupListener]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
             at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1343)
             at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
             at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1167)
             ... 11 more
     Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.springframework.web.context.ContextCleanupListener]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
             at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1353)
             at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1341)
             ... 13 more

   • Eventually, the following out of memory errors will appear:

     12-Jan-2016 23:17:51.834 INFO [Statistics Thread-MULTI_VM_PORTAL_CACHE_MANAGER-1] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
      java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.lang.OutOfMemoryError]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
     	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1353)
     	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1341)
     	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1206)
     	at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
     	at com.liferay.portal.module.framework.ModuleFrameworkClassLoader.loadClass(ModuleFrameworkClassLoader.java:98)
     	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
     	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:356)
     	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:349)
     	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:160)
     	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
     	at net.sf.ehcache.util.concurrent.ConcurrentHashMap.values(ConcurrentHashMap.java:2718)
     	at net.sf.ehcache.util.concurrent.ConcurrentHashMap.values(ConcurrentHashMap.java:220)
     	at net.sf.ehcache.statistics.extended.ExtendedStatisticsImpl$1.run(ExtendedStatisticsImpl.java:104)
     	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
     	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
     	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
     	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
     	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     	at java.lang.Thread.run(Thread.java:745)