Nexus Search: All shards failed. How to force fresh elasticsearch instance? - Sonatype Nexus Repository - Sonatype Community
Nexus Search: All shards failed. How to force fresh elasticsearch instance?
post by h9334fnv on Jun 13, 2022
I am currently running nexus 3.35.0 in docker. The nexus-data is in a CIFS Samba folder, bind-mounted via Docker.
Last weekend, I stopped and started it. Nexus still works, I can pull and push packages, docker images, etc. I can no longer use the search bar in the UI to search for a particular package. There is an error “all shards failed”.
The logs indicate that some indices are pending delete. I thought that maybe I could let Nexus start with a fresh instance of Elasticsearch, so I stopped it, waited for a minute, renamed the elasticsearch folder to a backup, then re-started nexus.
A fresh elasticsearch folder is created, but I am still getting the same errors, that the same indices are pending delete, and these never go away.
How can I force Nexus to use a fresh instance of elasticsearch? I have attached a copy of my logs indicating the pending index deletes as well as the “all shards failed” when I do a search on the UI.
2022-06-13 06:36:53,604+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#5]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [9e14d4f5d17bbef22424601413d6c650bc2ba074] still pending deletes present for shards [[9e14d4f5d17bbef22424601413d6c650bc2ba074], [9e14d4f5d17bbef22424601413d6c650bc2ba074][0]] - retrying
2022-06-13 06:36:54,001+0000 WARN [Timer-0] *SYSTEM java.util.prefs - Could not drop file-lock on user preferences. Unix error code 12.
2022-06-13 06:36:55,615+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#4]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [cd057c7addc9ab247d49640f034d8f63c9df6604] still pending deletes present for shards [[cd057c7addc9ab247d49640f034d8f63c9df6604], [cd057c7addc9ab247d49640f034d8f63c9df6604][0]] - retrying
2022-06-13 06:36:58,526+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#6]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [5541ebfd0a805671d170dd843afd2ffd71a03e16] still pending deletes present for shards [[5541ebfd0a805671d170dd843afd2ffd71a03e16], [5541ebfd0a805671d170dd843afd2ffd71a03e16][0]] - retrying
2022-06-13 06:37:27,804+0000 ERROR [qtp1751689652-115] admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Search.read, java-method: org.sonatype.nexus.coreui.SearchComponent.read
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:129)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:115)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:47)
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at org.sonatype.nexus.repository.search.query.SearchQueryServiceImpl.executeSearch(SearchQueryServiceImpl.java:211)
at org.sonatype.nexus.repository.search.query.SearchQueryServiceImpl.search(SearchQueryServiceImpl.java:157)
at org.sonatype.nexus.repository.search.query.SearchQueryService$search.call(Unknown Source)
at org.sonatype.nexus.coreui.SearchComponent.read(SearchComponent.groovy:107)
at com.palominolabs.metrics.guice.ExceptionMeteredInterceptor.invoke(ExceptionMeteredInterceptor.java:23)
at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:26)
at org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)
at org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
at org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native Method)
at DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeJavaMethod(DispatcherBase.invokeJavaMethod:142)
at org.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeJavaMethod(DispatcherBase.invokeJavaMethod:133)
at org.sonatype.nexus.extdirect.internal.ExtDirectDispatcher.invokeMethod(ExtDirectDispatcher.invokeMethod:82)
at org.softwarementors.extjs.djn.router.processor.standard.StandardRequestProcessorBase.dispatchStandardMethod(StandardRequestProcessorBase.dispatchStandardMethod:73)
at org.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequest(JsonRequestProcessor.processIndividualRequest:502)
at org.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequestsInThisThread(JsonRequestProcessor.processIndividualRequestsInThisThread:150)
at org.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.process(JsonRequestProcessor.process:133)
at org.sonatype.nexus.extdirect.internal.ExtDirectServlet.processRequest(ExtDirectServlet.java:632)
at org.sonatype.nexus.extdirect.internal.ExtDirectServlet.doPost(ExtDirectServlet.doPost:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.doServiceImpl:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.doServiceImpl:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.doServiceImpl:184)
at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.service:71)
at org.eclipse.jetty.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:85)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(OncePerRequestFilter.doFilter:66)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.executeChain:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilterInternal:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(AdviceFilter.doFilter:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilterInternal:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.executeChain:450)
at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.executeChain:96)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter$1.call:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.doCall:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.call:83)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilter(OncePerRequestFilter.doFilter:362)
at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.doFilterInternal:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.doFilter:116)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.doFilter:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.doFilter:79)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.doFilter:101)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.doFilter:98)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.doFilter:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doHandle(ServletHandler$Chain.doHandle:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.doHandle:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:143)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.nextHandle:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.doHandle:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.nextHandle:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(SessionHandler.doHandle:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.doHandle(ContextHandler.doHandle:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.doScope:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.doScope:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.doScope:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:141)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.doHandle:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:143)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.nextHandle:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.doHandle:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.nextHandle:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(SessionHandler.doHandle:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.doScope:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.doScope:1594)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
post by h9334fnv on Jun 19, 2022
Update: Was able to fix it by completely removing the elasticsearch folder instead of just renaming it.