Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-47099

Second node startup fails because it's unable to initialize scheduler

    Details

      Description

      To reproduce this issue you have to config the following environment:

      • Cluster
      • Sharding
      • Oracle
      • Scheduler Enabled

      Steps to reproduce:
      1- Start node 1
      2- After the complete start, start the node 2

      The second node suddenly shutdowns trying to get the scheduled jobs from the node one:

      10:14:02,811 INFO  [localhost-startStop-1][BaseDB:481] Database supports case sensitive queries
      10:14:03,391 INFO  [Incoming-2,LIFERAY-CONTROL-CHANNEL,ALBERTOCHAPARRO-7033][LicenseManager:?] Failed to find directory D:\servers\cluster\6.2\liferay-portal-6.2-ee-sp2-node2\data\license
      10:14:23,389 ERROR [localhost-startStop-1][MainServlet:209] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
      com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
      	at com.liferay.portal.events.StartupAction.run(StartupAction.java:74)
      	at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1238)
      	at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:206)
      	at javax.servlet.GenericServlet.init(GenericServlet.java:160)
      	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
      	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
      	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
      	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
      	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
      	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
      	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
      	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
      	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
      	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      Caused by: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
      	at com.liferay.portal.scheduler.ClusterSchedulerEngine.initialize(ClusterSchedulerEngine.java:244)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
      	at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:48)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
      	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
      	at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
      	at com.sun.proxy.$Proxy488.initialize(Unknown Source)
      	at com.liferay.portal.scheduler.SchedulerEngineHelperImpl.initialize(SchedulerEngineHelperImpl.java:574)
      	at com.liferay.portal.kernel.scheduler.SchedulerEngineHelperUtil.initialize(SchedulerEngineHelperUtil.java:237)
      	at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:198)
      	at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
      	at com.liferay.portal.events.StartupAction.run(StartupAction.java:68)
      	... 19 more
      Caused by: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to load scheduled jobs from cluster node ALBERTOCHAPARRO-51674
      	at com.liferay.portal.scheduler.ClusterSchedulerEngine.callMaster(ClusterSchedulerEngine.java:561)
      	at com.liferay.portal.scheduler.ClusterSchedulerEngine.initialize(ClusterSchedulerEngine.java:236)
      	... 53 more
      Caused by: java.util.concurrent.TimeoutException
      	at com.liferay.portal.kernel.cluster.FutureClusterResponses.get(FutureClusterResponses.java:88)
      	at com.liferay.portal.scheduler.ClusterSchedulerEngine.callMaster(ClusterSchedulerEngine.java:553)
      	... 54 more
      

      Technical Notes:
      During node 2 startup this server gets the node 1 ID executing a SELECT FOR UPDATE over lock_ table and it generates a lock on that table since the commit doesn't release it (just in Oracle). Then this node calls node 1 to get the scheduled jobs and the same query is executed in node 1 which waits until the lock is released so after ten seconds a timeout is triggered shutting down node 2.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  4 years, 7 weeks, 3 days ago

                  Packages

                  Version Package
                  6.1.X EE
                  6.2.X EE
                  7.0.0 M1