The "Reindex all spell check indexes" button will always miss companyId 0

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.0 DXP FP86, 7.0.X, 7.1.10.2 SP2, 7.1.10 DXP FP14, 7.1.X, 7.2.10 DXP GA1, 7.2.10 DXP FP2, 7.2.X, Master
• Fix Version/s: 7.0.0 DXP FP94, 7.0.10.15 DXP SP15, 7.0.X, 7.1.10 DXP FP19, 7.1.10.5 SP5, 7.1.X, 7.2.10 DXP FP7, 7.2.X, 7.3.4 CE GA5, 7.3.10 DXP GA1, Master
• Component/s: Search Infrastructure
• Labels:
  • QA-V
  • backend_test
  • ee-bp
  • ee-ts
  • liferay-dxp-70-sp15
  • liferay-dxp-71-sp5
  • liferay-fixpack-de-94-7010
  • liferay-fixpack-dxp-19-7110
  • liferay-fixpack-dxp-7-7210
• Branch Version/s:
  7.2.x, 7.1.x, 7.0.x
• Backported to Branch:
  Committed
• Fix Priority: 3
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/91275

Company-0 includes important indexes that allow for searching in the System Settings. If for some reason your indexes are corrupted or removed, the Reindex all and Reindex all spell check indexes button will not reindex for Company-0, meaning that your System Settings search will not return any results, at least in Liferay 7.0. You will have to manually reindex it by a Groovy script such as:

com.liferay.portal.kernel.search.IndexWriterHelperUtil.indexSpellCheckerDictionaries(0)
com.liferay.portal.kernel.search.IndexWriterHelperUtil.indexQuerySuggestionDictionaries(0)

The error that will be thrown is:

[2019-08-21T14:47:21,408][DEBUG][o.e.a.s.TransportSearchAction] [chris-1] [liferay-0][0], node[CDDZ2ZKDSXmvf4YwcWQFdA], [P], s[STARTED], a[id=AFubKnHgQQqNytUjkNwYxA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[liferay-0], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=false], types=[], routing='null', preference='null', requestCache=false, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"suggest":{"spellCheckRequest":{"text":"analytics","term":{"field":"spellCheckWord_en_US","size":1,"suggest_mode":"MISSING","accuracy":0.5,"sort":"SCORE","string_distance":"INTERNAL","max_edits":2,"max_inspections":5,"max_term_freq":0.01,"prefix_length":1,"min_word_length":4,"min_doc_freq":0.0}}}}}]
org.elasticsearch.transport.RemoteTransportException: [chris-1][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: no mapping found for field [spellCheckWord_en_US]
	at org.elasticsearch.search.suggest.SuggestionBuilder.populateCommonFields(SuggestionBuilder.java:307) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.suggest.term.TermSuggestionBuilder.build(TermSuggestionBuilder.java:452) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.suggest.SuggestBuilder.build(SuggestBuilder.java:175) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:841) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:637) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:596) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:387) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:126) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:359) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:355) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1107) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
[2019-08-21T14:47:21,409][DEBUG][o.e.a.s.TransportSearchAction] [chris-1] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: no mapping found for field [spellCheckWord_en_US]
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:657) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:131) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:220) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:174) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:220) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:463) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1114) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1226) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1200) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:367) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:361) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:355) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1107) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.2.jar:6.8.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
Caused by: java.lang.IllegalArgumentException: no mapping found for field [spellCheckWord_en_US]
	at org.elasticsearch.search.suggest.SuggestionBuilder.populateCommonFields(SuggestionBuilder.java:307) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.suggest.term.TermSuggestionBuilder.build(TermSuggestionBuilder.java:452) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.suggest.SuggestBuilder.build(SuggestBuilder.java:175) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:841) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:637) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:596) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:387) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:126) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:359) ~[elasticsearch-6.8.2.jar:6.8.2]
	... 9 more

Steps to Reproduce
The steps are inconsistent, but these are the closest I've found
1. Connect Liferay to a remote Elasticsearch instance. See https://help.liferay.com/hc/en-us/articles/360028711132-Installing-Elasticsearch#step-two-install-elasticsearch for steps
2. Go to your Elasticsearch Home's /data folder, and manually delete the /nodes folder. This will remove all indexes.
3. Restart Elasticsearch
4. Once ES is restarted, go to Liferay Control Panel and Reindex all search indexes and Reindex all spell check indexes.
5. Go to System Settings and search for a valid setting.
Expected: The search will return results
Actual: The search will return no results. I've only seen this happen in 7.0.
However, the point of this ticket isn't about search results, but that the reindexing buttons do not reindex Company-0 indexes.

To workaround this issue, I had to manually reindex Company-0 through a Groovy script (in 7.0)

com.liferay.portal.kernel.search.IndexWriterHelperUtil.indexSpellCheckerDictionaries(0)
com.liferay.portal.kernel.search.IndexWriterHelperUtil.indexQuerySuggestionDictionaries(0)

Testing
Master, 7.1.x, 7.2.x, always throw the above error, but still return search results.
7.0 - no search results are returned in System Settings.