Portal cache manager may not ready when received a notification from cluster

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: Master
• Fix Version/s: 7.2.10 DXP FP12, 7.2.X, 7.3.10 DXP FP1, 7.3.10.1 DXP SP1, 7.3.6 CE GA7, 7.3.X, Master
• Component/s: Fault Tolerance > Clustering Framework, Performance > Caching Infrastructure
• Labels:
  • ee-ts
  • liferay-dxp-73-sp1
  • liferay-fixpack-dxp-1-7310
  • liferay-fixpack-dxp-12-7210
• Branch Version/s:
  7.3.x, 7.2.x
• Backported to Branch:
  Committed
• Fix Priority: 3
• Git Pull Request: https://github.com/shuyangzhou/liferay-portal/pull/9605

Steps to Reproduce:

1. Prepare a Liferay Cluster with 2 nodes (nodeA + nodeB)
2. Start nodeA
   1. Navigate to Control Panel > Server Administration > Script
   2. Run the following script:

      import com.liferay.portal.kernel.service.UserLocalServiceUtil
      import com.liferay.portal.kernel.util.PortalUtil
      
      userId = PortalUtil.getUserId(actionRequest);
      
      (1..100).each {
          UserLocalServiceUtil.updateAgreedToTermsOfUse(userId, true)
          sleep 500
      }
      

3. Start nodeB
4. Observe logs in nodeB

Expected Result:
No errors are thrown

Actual Result:
IllegalStateExceptions are thrown

Sample stacktrace:

2020-11-16 21:38:03.185 ERROR [liferay/cache_replication-1][ParallelDestination:59] Unable to process message {destinationName=liferay/cache_replication, response=null, responseDestinationName=null, responseId=null, payload=[[email protected], values={companyId=0, clusterInvoke=false, groupId=0}}
com.liferay.portal.kernel.messaging.MessageListenerException: java.lang.IllegalStateException: Unable to get portal cache manager MULTI_VM_PORTAL_CACHE_MANAGER
    at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:32)
    at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
    at com.liferay.portal.messaging.internal.ParallelDestination$1.run(ParallelDestination.java:56)
    at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:752)
    at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:664)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Unable to get portal cache manager MULTI_VM_PORTAL_CACHE_MANAGER
    at com.liferay.portal.cache.multiple.internal.PortalCacheManagerUtil.getPortalCacheManager(PortalCacheManagerUtil.java:51)
    at com.liferay.portal.cache.multiple.internal.cluster.link.messaging.ClusterLinkPortalCacheClusterListener.handlePortalCacheClusterEvent(ClusterLinkPortalCacheClusterListener.java:69)
    at com.liferay.portal.cache.multiple.internal.cluster.link.messaging.ClusterLinkPortalCacheClusterListener.doReceive(ClusterLinkPortalCacheClusterListener.java:62)
    at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:26)