Uploaded image for project: 'PUBLIC - Liferay Portal Community Edition'
  1. PUBLIC - Liferay Portal Community Edition
  2. LPS-103224

Elasticsearch "Limit of total fields has been exceeded" error is produced if you have a lot of DDM Structures

Details

    • See PTR-1145
    • See PTR-1145
    • It is already handled by product team in PTR-1656

    Description


      Implemented solution

      7.3 version:

      7.0, 7.1 and 7.2 versions:

      • The implemented solution is included in:
        • 7.2: Fixpack 8 or Service Pack 3
        • 7.1: Fixpack 20 or Service Pack 5
        • 7.0: Fixpack 96 or Service Pack 15 + 1.2.0 version of Elasticsearch 6 connector
      • To avoid compatibility issues with existing Liferay installations, the new code will be disabled by default
      • In case you want to enable it, you will have to:
        1. Go to System Settings → Dynamic Data Mapping → Dynamic Data Mapping Indexer
        2. Deactivate "Enable Legacy Dynamic Data Mapping Index Fields." check
        3. Execute a full reindex

      If you have any custom development that execute queries in the Elasticsearch index using ddm __keyword __ and ddm __text __ fields, you will have to update your code and use the new nested ddmFieldArray document, see 7.3 version breaking changes: [https://github.com/liferay/liferay-portal/blob/7.3.x/readme/BREAKING_CHANGES.markdown#dynamic-data-mapping-fields-in-elasticsearch-have-changed-to-a-nested-document]


      Elasticsearch "Limit of total fields has been exceeded" error is produced in case of having a lot of DDM Structures in the Liferay installation.

      Following error is thrown in log file:

      2019-01-24 19:01:49.069 ERROR [liferay/search_writer/SYSTEM_ENGINE-423][ElasticsearchUpdateDocumentCommandImpl:50] failure in bulk execution:_[1]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_2557174], message [java.lang.IllegalArgumentException: Limit of total fields [7500] in index [liferay-20116] has been exceeded]_[3]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_2556975], message [java.lang.IllegalArgumentException: Limit of total fields [7500] in index [liferay-20116] has been exceeded] [Sanitized]
      

      The root cause of the issue is every time a new DDM Structure (webcontent, forms, document library) is created, each of its DDM fields are mapped in Elasticsearch to two new field mappings for each language.

      For example, in case you have a webcontent structure with 5 fields and you have 5 languages configured in your Liferay installation, in Elasticsearch side you will have:

      • 5 fields * 5 languages * 2 = 50 elasticsearch mappings

      Those new Elasticsearch mappings cannot be deleted until a full reindex is executed, so in case you create and delete several DDM Structures in a multi-language Liferay installation, it is easy to hit the Elasticsearch mapping limit.

      Steps to reproduce

      1. Create a Structure with a Text field
      2. Create a Webcontent using that structure
      3. Open Elasticsearch URL http://localhost:9201/_mapping?pretty and count the number of elasticsearch mappings (check the number of occurrences of "type" in the text)
      4. Create a new Structure with a Text field
      5. Create a Webcontent using that structure
      6. Open Elasticsearch URL http://localhost:9201/_mapping?pretty and count the number of elasticsearch mappings (check the number of occurrences of "type" in the text)
        • Expected behavior: The number of Elasticsearch mappings doesn't increase
        • Wrong behavior: The number of Elasticsearch mappings increases every time you create a new structure and content

      Workaround
      As a workaround, it is possible to increase the Elasticsearch mapping limit, but Elastic documentation warns having a large number of mappings is a bad practice that causes throughput issues, see:

      Attachments

        Issue Links

          Activity

            People

              brian.lee Brian Lee
              inacio.nery Inácio Nery (Inactive)
              Kiyoshi Lee Kiyoshi Lee
              André Ricardo Barreto de Oliveira André Ricardo Barreto de Oliveira
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                2 years, 16 weeks, 4 days ago

                Packages

                  Version Package
                  7.0.0 DXP FP96
                  7.0.10.15 DXP SP15
                  7.1.10 DXP FP20
                  7.1.10.5 SP5
                  7.2.10 DXP FP8
                  7.2.10.3 DXP SP3
                  7.3.10 DXP GA1
                  7.4.2 CE GA3 DXP 7,4
                  7.4.13 DXP GA1