Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-28638

LDAP failure to close NamingEnumeration causing thread locks

    Details

      Description

      LPS-26698 made some changes to ensure that NamingEnumeration instances were closed correctly. The impact of not having the NamingEnumeration instances close correctly was the failure to recover threads in the LDAP thread pool. Under circumstances where the LDAP server is not functioning correctly, this could eventually lead to server failure on an LDAP import or with repeated authentication where the system is unable to allocate new native threads.

      LPS-26698 missed correctly closing two NamingEnumeration instances (com.liferay.portal.security.ldap.PortalLDAPUtil.java SVN revision 107571). These need to be corrected as show in the attached patch file (for SVN revision 116403).

      For details on why this is causing a problem refer to http://blogs.warwick.ac.uk/kieranshaw/entry/ldap_connection_pooling/.

      To attempt to reproduce the fault, you can follow these steps (This is how the problem was originally detected in our test environment using a file version prior to 107571 - Actually as part of stress testing OpenSSO logins with open.sso.ldap.import.enabled=true).

      1. Include the following property in the portal-ext.properties file (The default value is 15 seconds):

      ldap.connection.com.sun.jndi.ldap.read.timeout=500

      2. Restart the portal and run a load test script that simply logs a user in and out repeatedly. Alternatively, configure the portal to do a LDAP import on start-up. Basically any action that will produce a large number of LDAP requests for data retrieved from LDAP.
      3. Check the log files for any LDAP read-timeout errors. If necessary, lower the value of the property and restart. Basically, our server was returning results slightly slowly at one point, resulting in timeout errors.
      4. Once you notice that there are a number of errors in the logs, allow the process to continue to run generating even more errors.

      Result:

      The failure to close the NamingEnumeration results in the connections to the LDAP server not being released correctly. As a result of the LDAP pool not being correctly configured (will create another issue for this), the number of LDAP connections grows unchecked. The number of connections can be seen to be growing as seen by:

      netstat -tcp -p | grep name_of_your_ldap_server

      You will also find that the number of threads locking in the JVM as a result of LDAP read requests increases (see thread dump). Eventually server failure will occur when no further native threads can be allocated.

      "tomcat-http--44" - Thread t@204
         java.lang.Thread.State: TIMED_WAITING
      	at java.lang.Object.wait(Native Method)
      	- waiting on <43723ee9> (a com.sun.jndi.ldap.LdapRequest)
      	at com.sun.jndi.ldap.Connection.readReply(Connection.java:452)
      	at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:611)
      	at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:534)
      	at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1965)
      	at com.sun.jndi.ldap.LdapCtx.doSearchOnce(LdapCtx.java:1914)
      	at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1307)
      	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:213)
      	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:121)
      	at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:133)
      	at com.liferay.portal.security.ldap.PortalLDAPUtil._getAttributes(PortalLDAPUtil.java:808)
      	at com.liferay.portal.security.ldap.PortalLDAPUtil.getGroupAttributes(PortalLDAPUtil.java:208)
      	at com.liferay.portal.security.ldap.PortalLDAPUtil.getGroupAttributes(PortalLDAPUtil.java:187)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterImpl.importGroup(PortalLDAPImporterImpl.java:757)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterImpl.importGroups(PortalLDAPImporterImpl.java:853)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterImpl.importLDAPUser(PortalLDAPImporterImpl.java:237)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterImpl.importLDAPUser(PortalLDAPImporterImpl.java:321)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterImpl.importLDAPUser(PortalLDAPImporterImpl.java:363)
      	at com.liferay.portal.security.ldap.PortalLDAPImporterUtil.importLDAPUser(PortalLDAPImporterUtil.java:65)
      	at com.liferay.portal.security.auth.OpenSSOAutoLogin.login(OpenSSOAutoLogin.java:153)
      	at com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.processFilter(AutoLoginFilter.java:192)
      :
      :
      :
         Locked ownable synchronizers:
      	- locked <641b9ae3> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              paul.piao Paul Piao (Inactive)
              Reporter:
              g.steyn Graeme Steyn
              Participants of an Issue:
              Recent user:
              Esther Sanz
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since last comment:
                8 years, 37 weeks, 4 days ago

                  Packages

                  Version Package
                  6.0.X EE
                  6.1.1 CE GA2
                  6.1.20 EE GA2
                  6.2.0 CE M3