In case of having a tomcat cluster with session replication, java.lang.ClassNotFoundException is thrown during session replication of classes that are loaded by a OSGI classloader

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.0 CE GA1, 7.0.0 DXP GA1, 7.0.X EE
• Fix Version/s: 7.0.0 DXP FP22, 7.0.0 DXP SP4, 7.0.X EE, 7.0.4 CE GA5, Master
• Component/s: Fault Tolerance > Clustering Framework, Portal Services > Component Management > Module Framework (OSGi)
• Labels:
  • backend_test
  • ee-ts
  • escalation
  • liferay-digital-enterprise-70-sp4
  • liferay-fixpack-de-22-7010
• Branch Version/s:
  7.0.x
• Backported to Branch:
  Committed
• Story Points: 1.5
• Fix Priority: 4
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/49692

In case of having a tomcat cluster with session replication, java.lang.ClassNotFoundException is thrown during session replication of classes that are loaded by a OSGI classloader:

02-Jun-2017 11:02:36.254 SEVERE [Tribes-Task-Receiver-1] org.apache.catalina.ha.session.DeltaManager.messageReceived Manager [localhost#]: Unable to receive message through TCP channel
 java.lang.ClassNotFoundException: test.session.replication.MySessionData
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
	at org.apache.catalina.tribes.io.ReplicationStream.resolveClass(ReplicationStream.java:80)

Root cause of this issue: tomcat, during session replication, has to deserialize replicated object, but at tomcat level there is no access to OSGI classloaders, so a ClassNotFoundException exception is thrown.

This issue can be reproduced:

• From webcontent portlet This portlet is using com.liferay.petra.collection.stack.FiniteUniqueStack class that is loaded using a classloader from a OSGI module
• From a external portlet that stores custom class inside portlet session, see attached example test-session-replication-portlet.war

Environment setup
Configure Liferay cluster with two tomcat nodes, see instructions in --LPS-72500--

Steps to reproduce using web content functionality

1. Start first tomcat node and wait until startup finish
2. Verify you can login from browser using URL: http://localhost:8080/web/guest
3. Start second tomcat node and wait until startup finish
4. Review LIFERAY_HOME/tomcat-8.0.32/logs/cluster.2017-xx-xx.log files of both nodes (see configuration explained in ----LPS-72500----) and verify that tomcat cluster is working.
5. Login in any node of cluster
6. Go to Control Panel => Web content section => create a new web content
7. Check cluster log files:
   • Correct behavior: No error is displayed
   • Wrong behavior: Following error is displayed in some session event replication, in the opposite node
     (if you create web content in first node, error is displayed in second one and vice-versa)

     02-Jun-2017 08:09:43.260 SEVERE [Tribes-Task-Receiver-2] org.apache.catalina.ha.session.DeltaManager.messageReceived Manager [localhost#]: Unable to receive message through TCP channel
      java.lang.ClassNotFoundException: com.liferay.petra.collection.stack.FiniteUniqueStack
     	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     	at java.lang.Class.forName0(Native Method)
     	at java.lang.Class.forName(Class.java:348)
     	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
     	at org.apache.catalina.tribes.io.ReplicationStream.resolveClass(ReplicationStream.java:80)
     	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
     	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
     	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
     	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
     	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
     	at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.readExternal(DeltaRequest.java:387)
     	at org.apache.catalina.ha.session.DeltaRequest.readExternal(DeltaRequest.java:277)
     	at org.apache.catalina.ha.session.DeltaManager.deserializeDeltaRequest(DeltaManager.java:565)
     	at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1233)
     

Steps to reproduce using 'test-session-replication-portlet.war'

1. Start first tomcat node and wait until startup finish
2. Verify you can login from browser using URL: http://localhost:8080/web/guest
3. Install attached test-session-replication-portlet.war in first tomcat node (copy war to liferay deploy directory)
4. Create a page called "session replication test"
5. Add test session replication portlet to created page
6. Start second tomcat node and wait until startup finish
7. Install attached test-session-replication-portlet.war in second tomcat node (copy war to liferay deploy directory)
8. Open created page "session replication test" in a new browser using URL of first node
9. Following message will be displayed: No sessionData exists, generated a new one with random string ....
10. Refresh page: generated random string will be displayed again
11. In same browser, open same page but in second node.
    • Correct behavior: Same random string generated in first node will be displayed in second one (random string is replicated through tomcat session replication)
    • Wrong behavior: A new random string is generated again (random string is not replicated through tomcat session replication)
12. Check cluster log files:
    • Correct behavior: No error is displayed
    • Wrong behavior: Following error is displayed in some session event replication, in the opposite node

      02-Jun-2017 10:52:11.871 SEVERE [Tribes-Task-Receiver-6] org.apache.catalina.ha.session.DeltaManager.messageReceived Manager [localhost#]: Unable to receive message through TCP channel
       java.lang.ClassNotFoundException: test.session.replication.MySessionData
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:348)
      	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:628)
      	at org.apache.catalina.tribes.io.ReplicationStream.resolveClass(ReplicationStream.java:80)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
      	at org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.readExternal(DeltaRequest.java:387)
      	at org.apache.catalina.ha.session.DeltaRequest.readExternal(DeltaRequest.java:277)
      	at org.apache.catalina.ha.session.DeltaManager.deserializeDeltaRequest(DeltaManager.java:565)
      	at org.apache.catalina.ha.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:1233)
      

---

If you install this test-session-replication-portlet.war in Liferay 6.2 version, exception is not reproduced and generated random string is replicated from one node to other thought session.

 

For QA, after fix, you need to set "portlet.session.replicate.enabled=true" in portal-ext.properties and same steps as above to verify.