Elasticsearch "Limit of total fields has been exceeded" error is produced if you have a lot of DDM Structures

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.X, 7.1.X, 7.2.X, Master
• Fix Version/s: 7.0.0 DXP FP96, 7.0.10.15 DXP SP15, 7.1.10 DXP FP20, 7.1.10.5 SP5, 7.2.10 DXP FP8, 7.2.10.3 DXP SP3, 7.3.10 DXP GA1, 7.4.2 CE GA3, 7.4.13 DXP GA1
• Component/s: Dynamic Data Mapping, Search Infrastructure
• Labels:
  • backend_test
  • ee-ts
  • liferay-dxp-70-sp15
  • liferay-dxp-71-sp5
  • liferay-dxp-72-sp3
  • liferay-fixpack-de-96-7010
  • liferay-fixpack-dxp-20-7110
  • liferay-fixpack-dxp-8-7210
  • liferay-ga4-ce-733
  • marketplace-portal-search-elasticsearch6-1.2.0-7010
  • marketplace-portal-search-elasticsearch7-3.1.0-7210
• Branch Version/s:
  7.2.x, 7.1.x, 7.0.x
• Backported to Branch:
  Committed
• Fix Priority: 4
• Sprint: Search | S03 Sprint 7, Search|S03 Sprint 19|7.21-8.10
• Git Pull Request: https://github.com/natocesarrego/liferay-portal/pull/722
• What problem did the customer find?: See PTR-1145
• What do you think is the root cause? (if known): See PTR-1145
• Proposed solution (if any): It is already handled by product team in PTR-1656

---

Implemented solution

7.3 version:

• In order to solve this issue, Dynamic Data Mapping fields in Elasticsearch that start with ddm __keyword __ and ddm __text __ have been moved to a new nested document ddmFieldArray, that avoids creating new Elasticsearch mappings everytime you create a ddm field in Liferay side.
• For more information about the solution, see the breaking changes here: https://github.com/liferay/liferay-portal/blob/7.3.x/readme/BREAKING_CHANGES.markdown#dynamic-data-mapping-fields-in-elasticsearch-have-changed-to-a-nested-document
• It is possible to restore old functionality activating the "Enable Legacy Dynamic Data Mapping Index Fields." check in System Settings → Dynamic Data Mapping → Dynamic Data Mapping Indexer.

7.0, 7.1 and 7.2 versions:

• The implemented solution is included in:
  • 7.2: Fixpack 8 or Service Pack 3
  • 7.1: Fixpack 20 or Service Pack 5
  • 7.0: Fixpack 96 or Service Pack 15 + 1.2.0 version of Elasticsearch 6 connector
• To avoid compatibility issues with existing Liferay installations, the new code will be disabled by default
• In case you want to enable it, you will have to:
  1. Go to System Settings → Dynamic Data Mapping → Dynamic Data Mapping Indexer
  2. Deactivate "Enable Legacy Dynamic Data Mapping Index Fields." check
  3. Execute a full reindex

If you have any custom development that execute queries in the Elasticsearch index using ddm __keyword __ and ddm __text __ fields, you will have to update your code and use the new nested ddmFieldArray document, see 7.3 version breaking changes: [https://github.com/liferay/liferay-portal/blob/7.3.x/readme/BREAKING_CHANGES.markdown#dynamic-data-mapping-fields-in-elasticsearch-have-changed-to-a-nested-document]

---

Elasticsearch "Limit of total fields has been exceeded" error is produced in case of having a lot of DDM Structures in the Liferay installation.

Following error is thrown in log file:

2019-01-24 19:01:49.069 ERROR [liferay/search_writer/SYSTEM_ENGINE-423][ElasticsearchUpdateDocumentCommandImpl:50] failure in bulk execution:_[1]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_2557174], message [java.lang.IllegalArgumentException: Limit of total fields [7500] in index [liferay-20116] has been exceeded]_[3]: index [liferay-20116], type [LiferayDocumentType], id [com.liferay.journal.model.JournalArticle_PORTLET_2556975], message [java.lang.IllegalArgumentException: Limit of total fields [7500] in index [liferay-20116] has been exceeded] [Sanitized]

The root cause of the issue is every time a new DDM Structure (webcontent, forms, document library) is created, each of its DDM fields are mapped in Elasticsearch to two new field mappings for each language.

For example, in case you have a webcontent structure with 5 fields and you have 5 languages configured in your Liferay installation, in Elasticsearch side you will have:

• 5 fields * 5 languages * 2 = 50 elasticsearch mappings

Those new Elasticsearch mappings cannot be deleted until a full reindex is executed, so in case you create and delete several DDM Structures in a multi-language Liferay installation, it is easy to hit the Elasticsearch mapping limit.

Steps to reproduce

1. Create a Structure with a Text field
2. Create a Webcontent using that structure
3. Open Elasticsearch URL http://localhost:9201/_mapping?pretty and count the number of elasticsearch mappings (check the number of occurrences of "type" in the text)
4. Create a new Structure with a Text field
5. Create a Webcontent using that structure
6. Open Elasticsearch URL http://localhost:9201/_mapping?pretty and count the number of elasticsearch mappings (check the number of occurrences of "type" in the text)
   • Expected behavior: The number of Elasticsearch mappings doesn't increase
   • Wrong behavior: The number of Elasticsearch mappings increases every time you create a new structure and content

Workaround
As a workaround, it is possible to increase the Elasticsearch mapping limit, but Elastic documentation warns having a large number of mappings is a bad practice that causes throughput issues, see:

• Mapping explosion: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings