Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-51428

ClusterForwardMessageListener attempts to send messages before portal start up completes

    Details

      Description

      Steps to reproduce in master:

      In a multiple node cluster if a node is restarted while the cluster is under load NullPointerExceptions occur.

      Example:

      11:51:50,892 ERROR [Incoming-1,LIFERAY-TRANSPORT-CHANNEL-0,g1t1950g-53961][NAKACK2:766] failed to deliver message [dst: <null>, src: g1t1953g-2219 (2 headers), size=1988 bytes]
      java.lang.NullPointerException
      	at com.liferay.portal.kernel.messaging.MessageBusUtil._sendMessage(MessageBusUtil.java:202)
      	at com.liferay.portal.kernel.messaging.MessageBusUtil.sendMessage(MessageBusUtil.java:90)
      	at com.liferay.portal.kernel.cluster.messaging.ClusterForwardMessageListener.receive(ClusterForwardMessageListener.java:42)
      	at com.liferay.portal.cluster.ClusterForwardReceiver.receive(ClusterForwardReceiver.java:44)
      	at org.jgroups.JChannel.invokeCallback(JChannel.java:749)
      	at org.jgroups.JChannel.up(JChannel.java:710)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
      	at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
      	at org.jgroups.protocols.UNICAST.up(UNICAST.java:414)
      	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:763)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
      	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
      	at org.jgroups.protocols.FD.up(FD.java:253)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
      	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
      	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
      	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1793)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      

      Steps to reproduce in 6.2.x:

      Steps to Reproduce
      1) Setup a 2 node cluster.
      2) Set these portal-ext properties:

      cluster.link.enabled=true
      lucene.replicate.write=true
      

      3) Start both nodes
      4) Deploy the create all things portlet to both nodes.
      5) On node 1, use the create all things portlet to create 10,000 users.
      6) While this process is running, restart the second node.

      Expected Result: The node will start without errors.
      Actual Result: The node starts with the errors

      11:51:50,892 ERROR [Incoming-1,LIFERAY-TRANSPORT-CHANNEL-0,g1t1950g-53961][NAKACK2:766] failed to deliver message [dst: <null>, src: g1t1953g-2219 (2 headers), size=1988 bytes]
      java.lang.NullPointerException
      	at com.liferay.portal.kernel.messaging.MessageBusUtil._sendMessage(MessageBusUtil.java:202)
      	at com.liferay.portal.kernel.messaging.MessageBusUtil.sendMessage(MessageBusUtil.java:90)
      	at com.liferay.portal.kernel.cluster.messaging.ClusterForwardMessageListener.receive(ClusterForwardMessageListener.java:42)
      	at com.liferay.portal.cluster.ClusterForwardReceiver.receive(ClusterForwardReceiver.java:44)
      	at org.jgroups.JChannel.invokeCallback(JChannel.java:749)
      	at org.jgroups.JChannel.up(JChannel.java:710)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
      	at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
      	at org.jgroups.protocols.UNICAST.up(UNICAST.java:414)
      	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:763)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
      	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
      	at org.jgroups.protocols.FD.up(FD.java:253)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
      	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
      	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
      	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1793)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              lu.liu Lu Liu
              Reporter:
              preston.crary Preston Crary
              Participants of an Issue:
              Recent user:
              Brian Wulbern
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                5 years, 21 weeks, 2 days ago

                  Packages

                  Version Package
                  6.1.X EE
                  6.2.4 CE GA5
                  6.2.X EE
                  7.0.0 M3