Two master nodes appear if master process hangs for a while

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 6.2.10 EE GA1, 6.2.X EE, 7.0.0 Alpha 1, 7.0.0 Alpha 2
• Fix Version/s: 6.1.X EE, 6.2.X EE, 7.0.0 Alpha 4
• Component/s: Fault Tolerance, Fault Tolerance > Clustering Framework
• Labels:
  • backend_test
  • ee-ts
  • liferay-fixpack-portal-83-6210
• Branch Version/s:
  6.2.x, 6.1.x
• Backported to Branch:
  Committed
• Story Points: 1
• Fix Priority: 2
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/32474

When the process of master node hangs for a while, another node becomes the master. Later, if the process of the original master node resumes, there will be two master nodes.

Steps to reproduce:

1. Install two Tomcat bundles.
2. Use the following simple portal-ext.properties for configuring basic clustering:

   jdbc.default.driverClassName=com.mysql.jdbc.Driver
   jdbc.default.url=jdbc:mysql://localhost/lportal?useUnicode=true&characterEncoding=UTF-8&useFastDateParsing=false
   jdbc.default.username=root
   jdbc.default.password=liferay
   
   cluster.link.enabled=true
   lucene.replicate.write=true
   

3. Start the nodes consecutively.
4. Run the following Groovy script in the nodes' Server Administration --> Script tab:

   import com.liferay.portal.kernel.cluster.ClusterMasterExecutorUtil;
   
   ClusterMasterExecutorUtil cmeu = new ClusterMasterExecutorUtil();
   out.println("master node?" + cmeu.isMaster());
   

   You get master node?true on master and master node?false on the other node.

5. Create heap dump from the first node using the command. (<JVM_PID> means the process id of Node 1 JVM.) This will make the process hang for a while.

   jmap -F -dump:file=heapdump-qa01.bin <JVM_PID>

6. Run the Groovy script again on both nodes.

Result: Both nodes return master node?true.
Expected result: One node returns master node?true and the other returns master node?false.

Alternative ways for suspending a process:

• Linux
  Suspend the process identifiedy by PID:

  kill -STOP PID

  Resume suspended process identifiedy by PID:

  kill -CONT PID

• Windows
  On windows you can suspend a process by using the Resource Monitor. (See screenshots pause1.png and pause2.png.)
  Also VisualVM might be useful to figure out which Java process you should suspend.