Japanese text is not properly being extracted from documents for indexing

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: Master
• Fix Version/s: 7.0.0 DXP SP2, 7.0.0 DXP FP13, 7.0.0 DXP SP3, 7.0.3 CE GA4, 7.1.X, Master
• Component/s: Search Infrastructure
• Labels:
  • backend_test
  • liferay-digital-enterprise-70-sp3
  • liferay-fixpack-de-13-7010
• Branch Version/s:
  7.0.x
• Backported to Branch:
  Committed
• Story Points: 1
• Fix Priority: 2
• Git Pull Request: https://github.com/brianchandotcom/liferay-portal/pull/47550

Steps to reproduce

1) Add attached text files to Document and Media
① en-ja-sjis-crlf-word.txt
② en-ja-sjis-crlf.txt
③ en-ja-sjis-lf-word.txt
④ en-ja-sjis-lf.txt
⑤ en-sjis-crlf.txt
⑥ en-sjis-lf.txt
⑦ ja-sjis-crlf-short.txt
⑧ ja-sjis-lf-short.txt

2) Go to a site page and search these two cases from the Search portlet:
Case 1:
Search for "東京"
Case 2:
Search for "ahead"

Expected behavior
Case 1:
①②③④⑦⑧ would be returned
Case 2:
①②③④⑤⑥ would be returned

Actual behavior
Case 1:
Only ②④ are returned in the results
Case 2:
Only ②④⑤⑥ are returned in the results

Reproduced on
master - e9ed60f