Second node startup fails because it's unable to initialize scheduler

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 6.1.20 EE GA2, 6.1.30 EE GA3, 6.2.10 EE GA1, 7.0.0 M3
• Fix Version/s: 6.1.X EE, 6.2.X EE, 7.0.0 M1
• Component/s: Core Infrastructure, Core Infrastructure > Sharding, Fault Tolerance > Clustering Framework, Portal Services > Scheduler
• Labels:
  • QA-NA
  • QA-TP
  • liferay-fixpack-portal-137-6210
  • liferay-fixpack-portal-51-6130
• Branch Version/s:
  6.2.x, 6.1.x
• Backported to Branch:
  Committed
• Story Points: 28
• Fix Priority: 5
• Databases:
  Oracle Database 11.1
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/19350

To reproduce this issue you have to config the following environment:

• Cluster
• Sharding
• Oracle
• Scheduler Enabled

Steps to reproduce:
1- Start node 1
2- After the complete start, start the node 2

The second node suddenly shutdowns trying to get the scheduled jobs from the node one:

10:14:02,811 INFO  [localhost-startStop-1][BaseDB:481] Database supports case sensitive queries
10:14:03,391 INFO  [Incoming-2,LIFERAY-CONTROL-CHANNEL,ALBERTOCHAPARRO-7033][LicenseManager:?] Failed to find directory D:\servers\cluster\6.2\liferay-portal-6.2-ee-sp2-node2\data\license
10:14:23,389 ERROR [localhost-startStop-1][MainServlet:209] com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
com.liferay.portal.kernel.events.ActionException: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
	at com.liferay.portal.events.StartupAction.run(StartupAction.java:74)
	at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1238)
	at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:206)
	at javax.servlet.GenericServlet.init(GenericServlet.java:160)
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to initialize scheduler
	at com.liferay.portal.scheduler.ClusterSchedulerEngine.initialize(ClusterSchedulerEngine.java:244)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
	at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:48)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:56)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
	at com.sun.proxy.$Proxy488.initialize(Unknown Source)
	at com.liferay.portal.scheduler.SchedulerEngineHelperImpl.initialize(SchedulerEngineHelperImpl.java:574)
	at com.liferay.portal.kernel.scheduler.SchedulerEngineHelperUtil.initialize(SchedulerEngineHelperUtil.java:237)
	at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:198)
	at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
	at com.liferay.portal.events.StartupAction.run(StartupAction.java:68)
	... 19 more
Caused by: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to load scheduled jobs from cluster node ALBERTOCHAPARRO-51674
	at com.liferay.portal.scheduler.ClusterSchedulerEngine.callMaster(ClusterSchedulerEngine.java:561)
	at com.liferay.portal.scheduler.ClusterSchedulerEngine.initialize(ClusterSchedulerEngine.java:236)
	... 53 more
Caused by: java.util.concurrent.TimeoutException
	at com.liferay.portal.kernel.cluster.FutureClusterResponses.get(FutureClusterResponses.java:88)
	at com.liferay.portal.scheduler.ClusterSchedulerEngine.callMaster(ClusterSchedulerEngine.java:553)
	... 54 more

Technical Notes:
During node 2 startup this server gets the node 1 ID executing a SELECT FOR UPDATE over lock_ table and it generates a lock on that table since the commit doesn't release it (just in Oracle). Then this node calls node 1 to get the scheduled jobs and the same query is executed in node 1 which waits until the lock is released so after ten seconds a timeout is triggered shutting down node 2.