This article addresses some of the common problems you may encounter when working with Elasticsearch 6, and serves as a troubleshooting guide to help you resolve them. Please notice that some of these problems apply only to Liferay DXP 7.0, while others apply to versions 7.0 and 7.1.
Resolution
Customers on Liferay DXP 7.0
-
Two Liferay Elasticsearch Adapters Are Active
Make sure you disable/deactivate the default Liferay Portal Search Elasticsearch (2.x adapter) module before switching, installing and configuring Liferay Connector to Elasticsearch 6. If you are doing it well, there should be only one Elasticsearch configuration available in the System Settings—called Elasticsearch 6—located under the Foundation section. Also, the Liferay Portal Search Elasticsearch module should be in Resolved state, but not Active in the App Manager.
-
Portal Search Elasticsearch (ES2) Adapter Started Again After Deleting the OSGi/State Folder
The state of the bundles is stored in the OSGi/State folder under your installation’s Liferay Home folder. If you delete this folder after patching, the Elasticsearch adapter is restarted automatically, and you must stop it manually. For more information on patching, please see the Customer Portal’s patching documentation.
-
NullPointerException When Deactivating Portal Search Elasticsearch (ES2) Adapter if Liferay Connector to Elasticsearch 6 is Already Installed and Active
If the default Elasticsearch 2 (ES2) adapter gets started, for whatever reason (see above), when the Elasticsearch 6 (ES6) adapter is already installed and active, you may get an NPE error when deactivating the ES2 adapter using the App Manager. As the ES2 adapter is not supposed to be active, the error can be ignored and does not indicate a problem. Here is a snippet of the error:
2018-04-06 15:34:35.000 ERROR [http-nio-8080-exec-5][com_liferay_portal_search:97] [com.liferay.portal.search.internal.SearchEngineHelperImpl(1575)] The removeSearchEngineConfigurator method has thrown an exception java.lang.NullPointerException at com.liferay.portal.search.elasticsearch.internal.connection.BaseElasticsearchConnection.getClusterHealthResponse(BaseElasticsearchConnection.java:80) at com.liferay.portal.search.elasticsearch.internal.connection.ElasticsearchConnectionManager.getClusterHealthResponse(ElasticsearchConnectionManager.java:81) at com.liferay.portal.search.elasticsearch.internal.ElasticsearchSearchEngine.waitForYellowStatus(ElasticsearchSearchEngine.java:341) at com.liferay.portal.search.elasticsearch.internal.ElasticsearchSearchEngine.initialize(ElasticsearchSearchEngine.java:112) at com.liferay.portal.kernel.search.SearchEngineProxyWrapper.initialize(SearchEngineProxyWrapper.java:95) at com.liferay.portal.search.internal.SearchEngineHelperImpl.setSearchEngine(SearchEngineHelperImpl.java:252) at com.liferay.portal.kernel.search.AbstractSearchEngineConfigurator.setSearchEngine(AbstractSearchEngineConfigurator.java:434) at com.liferay.portal.kernel.search.AbstractSearchEngineConfigurator.destroySearchEngine(AbstractSearchEngineConfigurator.java:232) at com.liferay.portal.kernel.search.AbstractSearchEngineConfigurator.destroy(AbstractSearchEngineConfigurator.java:93) at com.liferay.portal.search.elasticsearch6.internal.ElasticsearchEngineConfigurator.destroy(ElasticsearchEngineConfigurator.java:52) at com.liferay.portal.search.internal.SearchEngineHelperImpl.removeSearchEngineConfigurator(SearchEngineHelperImpl.java:303) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:224) at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39) at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:617) at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:501) at org.apache.felix.scr.impl.inject.BindMethod.invoke(BindMethod.java:655) at org.apache.felix.scr.impl.manager.DependencyManager.invokeUnbindMethod(DependencyManager.java:1837) at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeUnbindMethod(SingleComponentManager.java:394) at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:375) at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:291) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127) at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:917) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801) at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:222) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:908) at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:873)
-
LPKGVerifyException - Liferay Connector to 'Elasticsearch 6.lpkg' Does Not Have 'liferay-marketplace.properties' When Deploying
If you are seeing this error:
2018-07-13 09:31:26.740 ERROR [Start Level: Equinox Container: 00095840-7f86-0018-1430-b100bdbf75c2][DefaultLPKGDeployer:467] Unable to deploy LPKG file /opt/liferay-dxp-digital-enterprise-7.0/osgi/marketplace/Liferay Connector to Elasticsearch 6.lpkg com.liferay.portal.lpkg.deployer.LPKGVerifyException: com.liferay.portal.lpkg.deployer.LPKGVerifyException: /opt/liferay-dxp-digital-enterprise-7.0/osgi/marketplace/Liferay Connector to Elasticsearch 6.lpkg does not have liferay-marketplace.properties at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGVerifier.verify(DefaultLPKGVerifier.java:125) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer.deploy(DefaultLPKGDeployer.java:118) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer._instalLPKGs(DefaultLPKGDeployer.java:458) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer._activate(DefaultLPKGDeployer.java:341) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer.activate(DefaultLPKGDeployer.java:95)Double check that you have deployed the correct version of the Liferay Connector to ElasticSearch 6 application for your version of Liferay. If you need a previous version of the plugin, go back your Liferay Account's Purchased Apps page and click on Past Versions. For reference, here is the supported and compatibility chart for Liferay Enterprise Search.
-
LPKGVerifyException With Liferay Connector to Elasticsearch 6 in Fix Pack 42 (FP42) or Below
It is mandatory to install DE 7.0 Fix Pack 42 (FP42 = DE42) or higher to deploy and use Liferay Connector to Elasticsearch 6 on DXP DE 7.0 due to the API dependencies. Otherwise you’ll get a similar error upon portal startup:
06-Apr-2018 14:20:48.779 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class com.liferay.portal.spring.context.PortalContextLoaderListener java.lang.RuntimeException: com.liferay.portal.lpkg.deployer.LPKGVerifyException: LPKG validation failed with {[missing requirement com.liferay.portal.search.elasticsearch6.impl; version=1.0.1; type=osgi.bundle [caused by: Unable to resolve com.liferay.portal.search.elasticsearch6.impl version=1.0.1: missing requirement com.liferay.portal.search.suggest; version=[1.0.0,2.0.0)]]} at com.liferay.portal.spring.context.PortalContextLoaderListener.contextInitialized(PortalContextLoaderListener.java:262) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4812) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5255) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:585) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1794) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.liferay.portal.lpkg.deployer.LPKGVerifyException: LPKG validation failed with {[missing requirement com.liferay.portal.search.elasticsearch6.impl; version=1.0.1; type=osgi.bundle [caused by: Unable to resolve com.liferay.portal.search.elasticsearch6.impl version=1.0.1: missing requirement com.liferay.portal.search.suggest; version=[1.0.0,2.0.0)]]} at com.liferay.portal.lpkg.deployer.internal.LPKGIndexValidator.validate(LPKGIndexValidator.java:262) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer._activate(DefaultLPKGDeployer.java:333) at com.liferay.portal.lpkg.deployer.internal.DefaultLPKGDeployer.activate(DefaultLPKGDeployer.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) ... 2018-04-06 14:20:48.857 ERROR [localhost-startStop-1][PortalBeanLocatorUtil:109] BeanLocator is null 06-Apr-2018 14:20:48.858 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStop Exception sending context destroyed event to listener instance of class com.liferay.portal.spring.context.PortalContextLoaderListener com.liferay.portal.kernel.bean.BeanLocatorException: BeanLocator is not set at com.liferay.portal.kernel.bean.PortalBeanLocatorUtil.locate(PortalBeanLocatorUtil.java:74) at com.liferay.portal.spring.context.PortalContextLoaderListener.closeDataSource(PortalContextLoaderListener.java:381) at com.liferay.portal.spring.context.PortalContextLoaderListener.contextDestroyed(PortalContextLoaderListener.java:155) at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4859) at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5478) at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:159) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:585) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1794) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -
Need to Reindex All Spell Check Indexes After Upgrading to Fix Pack 42 (FP42) or Higher
Due to the removal of mapping types in Elasticsearch 6, administrators are required to execute Reindex all spell check indexes from the Server Administration to reindex spell check and query suggestion dictionaries even if still running on an Elasticsearch 2 server. See LPS-76675.
Customers on Liferay DXP 7.0 or 7.1
-
Unable to Access Kibana After Configuring To Work With Liferay Connector to X-Pack Monitoring
Once you set the
server.basePathinkibana.ymlyou cannot access the Kibana UI through https://localhost:5601. All access to the Kibana UI is via the monitoring portlet, which is only accessible to logged in DXP users. Navigate directly to the portlet using this URL: http://localhost:8080/o/portal-search-elasticsearch-xpack-monitoring/xpack-monitoring-proxy/app/monitoring. -
The Main Page Is Loaded in X-Pack Monitoring Portlet No Matter Which Tab the User Clicks on When Using Kibana 6.2.x
This is because the current integration is compatible with Elasticsearch 6.1.x and Kibana 6.1.x versions only. Please refer to the supported and compatibility chart for Liferay Enterprise Search to see the supported and compatible versions.
-
Possible ThreadLocal Leaking Upon Shutdown Caused by Usage of Netty in Elasticsearch 6
There is a known issue in Netty (network application framework) used by Elasticsearch internally. The following error messages may appear in the portal log upon stopping the application server/servlet container:
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@ccb08cb]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@21c8be8a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Even though there is a util method in Netty to do the proper clean-up the error message is still played after invoking it from a servlet listener and also through Tomcat's lifecycle listeners. This cleanup failure may be even a consequence of JVM's implementation details.
Because the Netty team has decided to simply live with the error, both Elasticsearch and Liferay will not be able to resolve this.
-
How to Disable X-Pack Security in Elasticsearch 6
1. Comment out any "xpack" related settings in your
elasticsearch.yml2. Add
xpack.security.enabled: falseSee more on the Elasticsearch site—Security Settings in Kibana: General Security Settings.