Search for categories in Japanese language is not working properly

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.1.X
• Fix Version/s: 7.1.X, 7.2.0 GA1
• Component/s: Search Infrastructure
• Labels:
  • QA-F
  • STAB_TST_JA
  • STAB_TST_JA_MUST
  • search-line-of-work-assigned
  • search-oct18
• Branch Version/s:
  7.1.x
• Backported to Branch:
  Committed
• Story Points: 0.25
• Fix Priority: 5
• Epic Link: Multi Language Search in 7.2
• Sprint: Search | S02 Sprint 3
• Git Pull Request: https://github.com/arboliveira/liferay-portal/pull/600

Fixed as part of LPS-84665

Summary
After creating a few Japanese language tags, and assigning those tags to Documents and Media assets, search does not work properly. The following example lists steps to reproduce in DM, but it happens in WCM as well.

Summary
Based on the behavior we have observed, it seems that tags are stored in a default index field which uses Standard analyzer which breaks down all Japanese Kanji characters into single characters. This is related to LPS-84665, but instead of Tags, relates to Categories.

Steps to Reproduce

1. Unzip vanilla Liferay bundle (clean, no portal-setup.wizard, etc.)
2. Initialize Liferay with Japanese as default language
3. Restart Server
4. After restarting, sign in with [e-mail] (you can use English for ease of use, i.e. append /en/ after hostname:port)
5. Navigate to Product Menu > Liferay DXP (Site) > Categorization > Categories
6. Click Add Vocabulary ボキャブラリ in the Japanese field
7. Click Add Category and add 東京 in Japanese (note: this is Tokyo)
8. Click Add Category and add 京都 in Japanese (note: this is Kyoto)
9. Navigate to Product Menu > Liferay DXP (Site) > Content > Web Content > Create a Basic Web Content
10. Add Web Content 豊島区 in Japanese, for both the title and Summary. Select 東京 Category in Metadata. (note: Toshima-ku is in Tokyo)
11. Add another Web Content 下京区 in Japanese, for both the title and Summary. Select 京都 Category in Metadata. (note: Shimogyo-ku is in Kyoto)
12. Change Display setting to Japanese (i.e. append /ja/ after hostname:port)
13. Return to the home page, and in the Search widget (top right corner), search for 東京

Actual Result
Even though we searched for 東京, search result includes 豊島区 and 下京区.

This is problematic because all of the results for 東京 should only include 豊島区, and NOT 下京区. In this case, the only commonality between 東京, 京都, and 下京区 is the character 京, so it seems that search is identifying these as search hits.

Expected Behavior
Results for 東京 should only include 豊島区 since this is the only categorization under 東京.

Reproduced in
7.1.x private Commit: 815320372a34faa0ccd0ed1d4989af7d1502c5e6