Liferay doesn't start correctly if remote Elasticsearch is not available or ES cluster is in red status

• Type: Bug
• Status: Closed
• Resolution: Fixed
• Affects Version/s: 7.0.X, 7.1.X, 7.2.X, Master
• Fix Version/s: 7.0.0 DXP FP97, 7.0.X, 7.1.10 DXP FP20, 7.1.10.5 SP5, 7.1.X, 7.2.X, 7.3.10 DXP FP1, 7.3.10.1 DXP SP1, Master
• Component/s: Core Infrastructure
• Labels:
  • ee-ts
  • liferay-dxp-71-sp5
  • liferay-dxp-73-sp1
  • liferay-fixpack-de-97-7010
  • liferay-fixpack-dxp-1-7310
  • liferay-fixpack-dxp-20-7110
  • liferay-fixpack-dxp-9-7210
• Branch Version/s:
  7.2.x, 7.1.x, 7.0.x
• Backported to Branch:
  Committed
• Fix Priority: 3
• Git Pull Request: https://github.com/shuyangzhou/liferay-portal/pull/9485

Liferay doesn't start correctly if remote Elasticsearch is not available or ES cluster is in red status.

The root cause of this problem is:

1. During Liferay startup, the instances and its search engines are initialized in: CompanyLocalServiceImpl.checkCompany => CompanyLocalServiceImpl.preregisterCompany methods
2. Them preregisterCompany method executes SearchEngineHelperUtil.initialize(companyId);
3. If Remote Elasticsearch is not available, that method throws an exception that aborts all the startup process done in checkCompany

---

This issue is very problematic for our customers because if your Elasticsearch data gets corrupted and you have to execute a reindex to solve the issue, you have to do it from Liferay interface, but you cannot access to the Liferay interface because Liferay cannot startup without the remote Elasticsearch server.

This problem is documented in several HelpCenter articles:

• https://help.liferay.com/hc/en-us/articles/360027091031-Blank-page-with-No-Group-exists-with-the-key-companyId-0-groupKey-Guest-error-in-log
• https://help.liferay.com/hc/en-us/articles/360044092272-Server-can-not-be-started-correctly-with-Unable-to-initialize-Elasticsearch-cluster-errors
• https://help.liferay.com/hc/en-us/articles/360042118652-Startup-failure-due-to-com-liferay-portal-kernel-exception-NoSuchRoleException-No-Role-exists-with-the-key-companyId-0-name-Owner-
• https://help.liferay.com/hc/en-us/articles/360046327752-External-Elasticsearch-server-is-not-available-and-control-panel-is-inaccessible
• https://help.liferay.com/hc/en-us/articles/360030923091-Liferay-DXP-does-not-start-after-configuring-Elasticsearch-connector-to-Remote-Operation-Mode

Workarounds:
There are two workarounds:
If your Elasticsearch server is working but it has corrupted data

1. Manually delete all data from your Elasticsearch server
2. Start Liferay
3. Execute a full reindex

If your Elasticsearch server is not available

1. Activate Embedded Elasticsearch using a config file, see https://help.liferay.com/hc/en-us/articles/360029031691-Elasticsearch-Connector-Settings-Reference
2. Start Liferay
3. Go to System Settings => Elasticsearch configuration and configure the new Remote Elasticsearch
4. Execute a full reindex

---

Reproduction steps Liferay 7.3 or master:
Due to some changes applied in LPS-108239, (see PortalInstances.java lines 281 to 287) this issue is only reproduced in master in case you are connecting to an empty Liferay database

1. Install a clean Liferay and point it to an empty database
2. Copy com.liferay.portal.search.elasticsearch7.configuration.ElasticsearchConfiguration.config file to LIFERAY_HOME/osgi/configs
3. Start Liferay
4. Several error traces will be displayed during startup
5. Try opening Liferay home page:
   • You can open Liferay home page, even if search functionality is not available
   • You cannot open Liferay home page

Reproduction steps Liferay 7.0, 7.1 and 7.2
The problem is reproduced in any Liferay installation

1. Start Liferay
2. Configure Remote Elasticsearch to connect to a fake Elasticsearch server, for example "example.com:9200"
3. Stop Liferay
4. Start Liferay
5. Several error traces will be displayed during startup
6. Try opening Liferay home page:
   • You can open Liferay home page, even if search functionality is not available
   • You cannot open Liferay home page