Quartz Sheduler initialization fails when sharding is actived and last initialized company is not in the default shard

Description

Steps to reproduce:

  • Configure 2 shards (default, one) and start portal

  • Quartz tables are properly created on default DB

  • Create a portal instance on shard one

  • Restart the portal

Observed behavior:

  • Following exception is thrown

    19:03:56,815 ERROR [BasePortalLifecycle:45] com.liferay.portal.kernel.scheduler.SchedulerException: Unable to start scheduler com.liferay.portal.kernel.scheduler.SchedulerException: Unable to start scheduler at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngine.start(QuartzSchedulerEngine.java:430) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.liferay.portal.kernel.util.MethodHandler.invoke(MethodHandler.java:83) at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:57) at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:50) at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:65) at com.liferay.portal.kernel.messaging.sender.DirectSynchronousMessageSender.send(DirectSynchronousMessageSender.java:68) at com.liferay.portal.kernel.messaging.sender.DefaultSingleDestinationSynchronousMessageSender.send(DefaultSingleDestinationSynchronousMessageSender.java:41) at com.liferay.portal.messaging.proxy.MessagingProxyAdvice.doInvokeSynchronous(MessagingProxyAdvice.java:106) at com.liferay.portal.messaging.proxy.MessagingProxyAdvice.invoke(MessagingProxyAdvice.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy470.start(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112) at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:56) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211) at $Proxy471.start(Unknown Source) at com.liferay.portal.kernel.scheduler.SchedulerEngineUtil._start(SchedulerEngineUtil.java:737) at com.liferay.portal.kernel.scheduler.SchedulerEngineUtil.start(SchedulerEngineUtil.java:260) at com.liferay.portal.kernel.scheduler.SchedulerLifecycle.doPortalInit(SchedulerLifecycle.java:30) at com.liferay.portal.kernel.util.BasePortalLifecycle.portalInit(BasePortalLifecycle.java:42) at com.liferay.portal.kernel.util.PortalLifecycleUtil.flushInits(PortalLifecycleUtil.java:45) at com.liferay.portal.servlet.MainServlet.initPlugins(MainServlet.java:815) at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:212) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:992) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4058) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4371) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:627) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1149) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:578) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) Caused by: org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.JobPersistenceException: Couldn't recover jobs: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist [See nested exception: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist]] at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:680) at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:518) at org.quartz.impl.StdScheduler.start(StdScheduler.java:143) at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngine.start(QuartzSchedulerEngine.java:423) ... 73 more Caused by: org.quartz.JobPersistenceException: Couldn't recover jobs: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist [See nested exception: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist] at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:886) at org.quartz.impl.jdbcjobstore.JobStoreSupport$1.execute(JobStoreSupport.java:824) at org.quartz.impl.jdbcjobstore.JobStoreSupport$46.execute(JobStoreSupport.java:3773) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3807) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3769) at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:820) at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:678) ... 76 more Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:936) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723) at com.mysql.jdbc.Connection.execSQL(Connection.java:3256) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1500) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1485) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.updateTriggerStatesFromOtherStates(StdJDBCDelegate.java:259) at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:841) ... 82 more 19:03:56,819 ERROR [MainServlet:350] java.lang.IllegalStateException: Unable to initialize portal java.lang.IllegalStateException: Unable to initialize portal at com.liferay.portal.kernel.util.BasePortalLifecycle.portalInit(BasePortalLifecycle.java:47) at com.liferay.portal.kernel.util.PortalLifecycleUtil.flushInits(PortalLifecycleUtil.java:45) at com.liferay.portal.servlet.MainServlet.initPlugins(MainServlet.java:815) at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:212) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1172) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:992) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4058) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4371) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:627) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1149) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:578) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) Caused by: com.liferay.portal.kernel.scheduler.SchedulerException: Unable to start scheduler at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngine.start(QuartzSchedulerEngine.java:430) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.liferay.portal.kernel.util.MethodHandler.invoke(MethodHandler.java:83) at com.liferay.portal.kernel.messaging.proxy.ProxyRequest.execute(ProxyRequest.java:57) at com.liferay.portal.kernel.messaging.proxy.ProxyMessageListener.receive(ProxyMessageListener.java:50) at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:65) at com.liferay.portal.kernel.messaging.sender.DirectSynchronousMessageSender.send(DirectSynchronousMessageSender.java:68) at com.liferay.portal.kernel.messaging.sender.DefaultSingleDestinationSynchronousMessageSender.send(DefaultSingleDestinationSynchronousMessageSender.java:41) at com.liferay.portal.messaging.proxy.MessagingProxyAdvice.doInvokeSynchronous(MessagingProxyAdvice.java:106) at com.liferay.portal.messaging.proxy.MessagingProxyAdvice.invoke(MessagingProxyAdvice.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy470.start(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:112) at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:56) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ChainableMethodAdvice.invoke(ChainableMethodAdvice.java:59) at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:108) at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:211) at $Proxy471.start(Unknown Source) at com.liferay.portal.kernel.scheduler.SchedulerEngineUtil._start(SchedulerEngineUtil.java:737) at com.liferay.portal.kernel.scheduler.SchedulerEngineUtil.start(SchedulerEngineUtil.java:260) at com.liferay.portal.kernel.scheduler.SchedulerLifecycle.doPortalInit(SchedulerLifecycle.java:30) at com.liferay.portal.kernel.util.BasePortalLifecycle.portalInit(BasePortalLifecycle.java:42) ... 30 more Caused by: org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.JobPersistenceException: Couldn't recover jobs: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist [See nested exception: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist]] at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:680) at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:518) at org.quartz.impl.StdScheduler.start(StdScheduler.java:143) at com.liferay.portal.scheduler.quartz.QuartzSchedulerEngine.start(QuartzSchedulerEngine.java:423) ... 73 more Caused by: org.quartz.JobPersistenceException: Couldn't recover jobs: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist [See nested exception: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist] at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:886) at org.quartz.impl.jdbcjobstore.JobStoreSupport$1.execute(JobStoreSupport.java:824) at org.quartz.impl.jdbcjobstore.JobStoreSupport$46.execute(JobStoreSupport.java:3773) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3807) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3769) at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:820) at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:678) ... 76 more Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'lportal_trunk_one.QUARTZ_TRIGGERS' doesn't exist at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:936) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2985) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723) at com.mysql.jdbc.Connection.execSQL(Connection.java:3256) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1585) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1500) at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1485) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.updateTriggerStatesFromOtherStates(StdJDBCDelegate.java:259) at org.quartz.impl.jdbcjobstore.JobStoreSupport.recoverJobs(JobStoreSupport.java:841) ... 82 more

    Desired behavior:

  • Persisted quartz jobs should be read from the default DB

Environment

MySQL

Activity

Show:

Cynthia Wilburn March 12, 2012 at 2:11 PM

Reopening to add 6.1.1 CE GA2. Close as Fixed.

Sophia.Zhang February 13, 2012 at 9:51 PM
Edited

PASSED Manual Testing following the steps in the description.

Reproduced on:
Tomcat 7.0 + MySQL 5. 6.1.0GA1.

When restart portal, above errors display in console.

Fixed on:

Tomcat 7.0.25 + MySQL 5. 6.1.x GIT ID: 9fbc284ff2c4b5102647c3640c4ddbff2736f30d.
Tomcat 7.0.25 + MySQL 5. 6.2.x GIT ID: 00cb08388ab375e2b5714809057c07fa7e1f67a6.

No above errors display in console anymore.

Michael Saechang February 10, 2012 at 2:35 PM

Committed on:
6.1.x GIT ID: cfb66214fb54a3c47d8d81838a884dbb8e276c83.
6.2.x GIT ID: cfb66214fb54a3c47d8d81838a884dbb8e276c83.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Branch Version/s

6.1.x

Backported to Branch

Committed

Affects versions

Priority

Zendesk Support

Created February 1, 2012 at 9:06 AM
Updated June 24, 2023 at 3:56 PM
Resolved March 12, 2012 at 2:37 PM