Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-75134

S3Store leaks HTTP connections when serving files as stream

    Details

      Description

      Description
      When all connections are leaked, S3Store stops working as connection pool keeps all threads waiting. As a result, Document Library stops working.

      Steps to Reproduce (via Groovy script)

      1. Setup Liferay DXP with S3Store configured
      2. Navigate to Documents and Media
        1. Create a Basic Document
      3. Navigate to Server Administration -> Script -> Groovy
        1. Copy/Paste the attached groovy script: test_s3_file_stream.groovy
      4. Execute Script (At least 3 times)
        • By default, the maximum¬†HTTP connections is set to 50.
        • Each time the script runs, it will make 20 requests for retrieving the document as a stream.
          • Running the script 3 times would make 60 requests total.

      Expected Results
      There will be no errors and the HTTP connections will not be exhausted

      Actual Results
      A ConnectionPoolTimeoutException is thrown:

      com.liferay.portal.kernel.exception.SystemException: Unable to execute HTTP request: Timeout waiting for connection from pool
      	at com.liferay.portal.store.s3.S3Store.transform(S3Store.java:809)
      	at com.liferay.portal.store.s3.S3Store.getS3ObjectSummaries(S3Store.java:642)
      	at com.liferay.portal.store.s3.S3Store.getHeadVersionLabel(S3Store.java:552)
      	at com.liferay.portal.store.s3.S3Store.getS3Object(S3Store.java:584)
      	at com.liferay.portal.store.s3.S3Store.getFileAsStream(S3Store.java:203)
              ...
      Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:642)
      	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:400)
      	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:362)
              ...
      Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      	at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)
      	at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
      

      Additional Info
      From the source code, we can observe that when we request file to be returned as a stream, it provides the actual InputStream from the S3Object.
      Reference: S3Store.java#L205

      This means that if the requested stream is not closed, it could cause a potential connection leak.

        Attachments

          Activity

            People

            Assignee:
            lu.liu Lu Liu
            Reporter:
            eric.yan Eric Yan
            Participants of an Issue:
            Recent user:
            Csaba Turcsan
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since last comment:
              3 years, 45 weeks, 4 days ago

                Packages

                Version Package
                7.0.0 DXP FP33
                7.0.0 DXP SP7
                7.0.5 CE GA6
                7.0.X
                7.1.X
                Master