Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-117254

When over 10,000 results are available for User Groups, pages beyond 500 are duplicates

    Details

    • Fix Priority:
      4

      Description

      The root cause of this problem is connected with LPS-72400. When trying to retrieve records beyond the index.max_result_window, ES throws an error and seems to only return results from before 10,000. Liferay ingests these results and returns them as the last results. So if results 10,000-10,020 should be shown on page 501, it will only display 9,980-10,000 which is the same as page 500.

      Furthermore, ES7 is using a new parameter "track_total_hits" that needs to be set in order to return accurate result numbers after 10,000 results. See: [reference|
      https://www.elastic.co/guide/en/elasticsearch/reference/7.0/search-request-track-total-hits.html#search-request-track-total-hits]. So in liferay versions after ES7, liferay acts as if there are no more than 10,000 results and will only paginate to page 500.

      Steps to Reproduce:

      1. Go to Control Panel > Configuration > Server Administration > Script. Paste and execute this groovy script to create 10K+ User Groups. (It took about 7 min for the user groups to be created)
        import com.liferay.portal.kernel.model.User;
        import com.liferay.portal.kernel.service.UserGroupLocalServiceUtil;
        import com.liferay.portal.kernel.service.ServiceContext;
        import com.liferay.portal.kernel.service.ServiceContextFactory;
        import com.liferay.portal.kernel.service.UserLocalServiceUtil;
        import com.liferay.portal.kernel.util.StringPool;
        
        int numUserGroups = 10001;
        
        String userGroupName = "Test User Group ";
        
        try {
        	ServiceContext serviceContext = ServiceContextFactory.getInstance(
        			actionRequest);
        
        	long userId = serviceContext.getUserId();
        
        	User user = UserLocalServiceUtil.getUser(userId);
        
        	long companyId = user.getCompanyId();
        
        	for (int i = 1; i <= numUserGroups; i++) {
        		String name = userGroupName + i;
        
        		UserGroupLocalServiceUtil.addUserGroup(
        				userId, companyId, name, StringPool.BLANK, serviceContext);
        
        		if (i % 100 == 0) {
        			System.out.println("Created " + i + " User Groups");
        		}
        	}
        }
        catch (Exception e) {
        	println("Exception Thrown: " + e);
        }
        
      2. Go to Control Panel > Configuration > System Setting > Foundation > Elasticsearch. Uncheck "Log exceptions only" and Save.
      3. Go to Control Panel > Users > User Groups
      4. Click on page 501 of the search container (note )

      Expected Result:
      Click on page 501 of the search container, no errors in console, user group "Test User Group 10001" is on that page

      Actual Result:
      In versions of liferay using ES 7+, such as master, page 501 doesn't exist. In older versions clicking on page 501 has the same results as page 500.
      An exception is thrown in the logs:

      2020-07-15 23:21:57.209 ERROR [http-nio-8080-exec-4][ElasticsearchIndexSearcher:173] java.lang.RuntimeException: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed; shardFailures {[hTasRcRSSoGcoYI739cSbQ][liferay-20101][0]: RemoteTransportException[[hTasRcR][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]; }
      java.lang.RuntimeException: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed; shardFailures {[hTasRcRSSoGcoYI739cSbQ][liferay-20101][0]: RemoteTransportException[[hTasRcR][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]; }
      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293)
      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133)
      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254)
      	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101)
      	at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48)
      	at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222)
      	at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176)
      	at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48)
      	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222)
      	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)
      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
      	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:464)
      	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1130)
      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1247)
      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1221)
      	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66)
      	at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:112)
      	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:347)
      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:341)
      	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
      	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
      	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      This exception comes from this call which creates a query with size > 10000.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              support-lep@liferay.com SE Support
              Reporter:
              joshua.cords Joshua Cords
              Participants of an Issue:
              Recent user:
              Tibor Lipusz
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Days since last comment:
                1 year, 10 weeks, 1 day ago

                  Packages

                  Version Package