Scheduler jobs are lost if master token released transition fails

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 6.2.X EE, 7.0.X EE, Master
• Fix Version/s: 6.2.X EE, 7.0.0 DXP FP30, 7.0.4 CE GA5, 7.1.X, Master
• Component/s: Fault Tolerance > Clustering Framework
• Labels:
  • backend_test
  • ee-ts
  • liferay-digital-enterprise-70-sp5
  • liferay-fixpack-de-30-7010
  • liferay-fixpack-portal-164-6210
• Branch Version/s:
  6.2.x
• Backported to Branch:
  Committed
• Story Points: 1.75
• Fix Priority: 3
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/51198

Steps to Reproduce:

1. Start a cluster with a couple of nodes
2. Simulate a connection lost between both nodes, so that both become master.
3. Recreate the connection, but make master answer node last more than clusterable.advice.call.master.timeout seconds (i.e. by adding a breakpoint in com.liferay.portal.scheduler.multiple.internal.ClusterSchedulerEngine.getScheduledJobs(StorageType))
4. Stop master node.
5. Execute the following code in the only available node:

   import com.liferay.portal.kernel.scheduler.*;
   import com.liferay.portal.kernel.scheduler.StorageType;
   def schedulerJobs = SchedulerEngineHelperUtil.getScheduledJobs(StorageType.MEMORY_CLUSTERED);
   out.println(schedulerJobs.size());
   

Actual Results:

Scheduled memory clustered jobs are empty.

Expected Results:

Even if an error occurs, jobs shouldn't have lost, maybe not fully synchronized, but at least not lost.