Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-43399

Concurrent file upload/update/deletion in Document library portlet is slow due to row lock contention

    Details

      Description

      Steps to reproduce

      • Create a folder in DL called Test.
      • Upload a single 1-byte text file (test.txt) to folder Test with 50 parallel user sessions. Each user session should upload the file 100 times and then modify 5 times. Thus, you should have 5000 files in folder Test after the whole process finishes.
      • After the test has started, monitor what the DB sessions are doing during the upload with the following SQL.
      SELECT x.session_id,
        x.event,
        ROUND(x.time_waited / x.total_time_waited * 100, 2) AS time_waited_pct
      FROM
        (SELECT t.*,
          SUM(t.time_waited) OVER (PARTITION BY t.session_ID) AS total_time_waited
        FROM
          (SELECT ash.session_id,
            se.event,
            SUM(se.time_waited) AS time_waited
          FROM v$active_session_history ash,
            v$session s,
            v$session_event se
          WHERE ash.sql_id   = 'gymh3tffcd0x6'
          AND ash.session_id = s.sid
          AND s.sid          = se.sid
          AND se.time_waited > 0
          GROUP BY ash.session_id,
            se.event
          ) t
        ) x
      

      Checkpoint: You should see that vast majority of wait time (>90%) will be spent on event enq: TX - row lock contention, which means that that record in table DLFolder which represents folder Test is being updated concurrently. (See sample output in the attached excel sheet.)

      Analysis

      In methods addFileEntry() and updateFileEntry() of class DLFileEntryLocalServiceImpl the following piece of code is responsible for causing this lock contention.

      // Folder
      
      if (folderId != DLFolderConstants.DEFAULT_PARENT_FOLDER_ID) {
      	DLFolder folder = dlFolderPersistence.findByPrimaryKey(folderId);
      
      	folder.setLastPostDate(fileEntry.getModifiedDate());
      
      	dlFolderPersistence.update(folder, false);
      }
      

      As you can see, records in DLFolder are updated upon every file entry change, thus serializing concurrent requests.

      Next Actions

      Updates to DLFolder.lastPostDate will be made asynchronous.

      Notes for QA

      • There are basically two ways to verify that the issue has been fixed indeed.
      • One is that you can see above (A), which needs some preparation and an easier one (B).
      • If you choose (A), on trunk you'll see that lock contention takes only ~50% of query execution time, whilst on 6.0.x it's much worse ~95%. This means that if you see "only" ~50%, you still reproduced the issue.

      Easy steps to reproduce (B)

      • Making the update of lastPostDate asynchronous means that this action is handled by a background thread.
      • Having this in mind you'll have to verify that.
      • You don't necessarily have to use Oracle, you can use MySQL as well.

      1) Create a folder Test, look up its ID in the DB.
      2) Execute the following query in order to lock that row in the DB which represents folder Test.

      -- The first statement is for MySQL only
      START TRANSACTION;
      -- Lock that row
      SELECT * FROM DLFolder WHERE folderId = <ID> FOR UPDATE;
      

      3) Upload a document to folder Test.

      Checkpoint: You should see that the document has been uploaded instantly to that folder, which means that even though the corresponding row of folder Test is locked in the DB, updating its last post date isn't happening right at that time when you upload something to it; instead it takes place in the background.

      4) Take a thread dump and you'll see that one thread is hanging, because it isn't able to update that row, because you placed an explicit lock on it.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lu.liu Lu Liu
                Reporter:
                laszlo.csontos Laszlo Csontos (Inactive)
                Participants of an Issue:
                Recent user:
                Brian Wulbern
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Days since last comment:
                  4 years, 40 weeks ago

                  Packages

                  Version Package
                  6.2.3 CE GA4
                  6.2.X EE
                  7.0.0 M1