Failure servicing: PUT ArrayIndexOutOfBoundsException

post by martin.schwamberger on Aug 20, 2020

Hello together,

We have an issue that the maven deploy task fails quite often with an error 500.

As is can see, these files are accessible in our nexus repositories, so the actual file deployment works. But it seems as there are some background tasks which are failing and therefore reporting an 500.

Our nexus instance is running and available, so the 500 is confusing in some ways.

This error effects not only the deployment, but sometimes fetching dependencies are also failing with this 500 error. We have to run the deploy / fetch task up to ten times until it succeeds.

We disabled all tasks defined in nexus tasks and tried a database cleanup on a cloned instance.

For the database cleanup:

The database check reveals some RIDs for non existing assets. After we removed the entries which references to these RIDs and deleted the indexes to be rebuild, the database check succeeds.

Unfortunately the database repair reveals even more RIDs for which we cannot find any reference to. So we cannot delete anything and the repair fails.

Despite the minor cleanup, this issue is still occurring (on the cloned test instance).

The nexus log files reveal on every issue with the same error: ArrayIndexOutOfBoundsException

In this example it is for PUT and “error during record deletion”, but it also happens with “error on record creation”.

Does anyone have an idea where this is from? And how we can fix it? Maybe even how to avoid this in the future?
Thank you

2020-08-19 00:20:10,749+0000 WARN [qtp366099936-8751] deployment org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT /repository/ld-public-snapshots/de/my/example/plugins/example-plugin/1.19.3-SNAPSHOT/example-plugin-1.19.3-20200819.002010-8.jar.md5
com.orientechnologies.orient.core.exception.OPaginatedClusterException: Error during record deletion
DB name=“component”
Component Name=“asset”
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.deleteRecord(OPaginatedCluster.java:944)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:3978)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedCluster.java:1857)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:182)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
at org.sonatype.nexus.repository.maven.internal.orient.MavenFacetImpl.put(MavenFacetImpl.java:305)
at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.doPut(HostedHandler.java:92)
at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.handle(HostedHandler.java:64)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.storage.LastDownloadedHandler.handle(LastDownloadedHandler.java:59)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:46)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.maven.internal.orient.OrientVersionPolicyHandler.handle(OrientVersionPolicyHandler.java:69)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.clm.internal.FirewallContributedHandler.handle(FirewallContributedHandler.java:98)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:42)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.routing.internal.RoutingRuleHandler.handle(RoutingRuleHandler.java:52)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:213)
at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)  
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChainInvocation:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilter(OncePerRequestFilter.doFilter(FilterChainInvocation.doFilter(OncePerRequestFilter.doFilter(FilterChainInvocation.doFilter(AdviceFilter.doFilter(AdviceFilter.doFilter(OncePerRequestFilter.doFilterOncePerRequestFilter.doFilterAdviceFilter.doFilterFilter(AdviceFilter.executeChain(AdviceFilter.filterFilter(FilterChain.executeAdvice(FilterAdviceFilter.doFilter(AdviceFilter.executeChain(AdviceFilter.doFilter(AdviceFilter.doFilter(adviceFilter: Yet(Anyway, Essentially, Someone will use this help to access the document DBA dies and someone else should make sure to lookup what is going on in the next few sections.<br>

DB name="component"
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:465) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:407) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:113) at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPositionMap.remove(OClusterPositionMap.java:484) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.deleteRecord(OPaginatedCluster.java:934) ... 138 common frames omitted

## post by luuk.peters_github on Feb 9, 2021

Martin, did you eventually resolve this issue?

Our nexus instance shows the same error in the log. Not every push fails, so the workaround in our case is to just try the push again, which is frustrating.

## post by mpiggott on Feb 9, 2021

This is likely some sort of database corruption.

## post by martin.schwamberger on Feb 10, 2021

Yes, we solved it with some losses.

We made a database export with cleanup and repair commands.

I cant find the exact commands we used, but you should find them on multiple posts about database repairing.

I recommend to do a backup and test these actions on a cloned instance before running in prod.

We decided that the few artifacts we lose can easily be reuploaded by us. Due to the corruption we could not find any other way to fix this and keep everything. So it was better to make a cut now and prevent even more data going corrupt.

If anyone find a way to fix this easily, I’m still listening for future problems

## post by mpiggott on Feb 10, 2021

We’re currently working on replacing OrientDB in the near future with better alternatives.

## post by luuk.peters_github on Feb 19, 2021

We applied the database export method as well. During the export, the erroneous records were skipped automatically by the tool. The component database itself is smaller than the actual blob store, which meant we could easily copy it for test purposes. So we tried the method on a copy of the database and verified by running the `CHECK DATABASE` command in the nexus orient console. Before the fix, a lot of stacktraces were showing, which were all gone after the fix.

So we set out applying the fix on the actual database, stopping the nexus service first of course.

```javascript
java -jar /opt/nexus/nexus-3.29.2-02/lib/support/nexus-orient-console.jar
connect plocal:/opt/nexus/sonatype-work/nexus3/db/component/ admin admin
Reran `CHECK DATABASE` to verify the issue is still present, which it was.

Executing:
export database component-export
drop database
create database plocal:/opt/nexus/sonatype-work/nexus3/db/component
import database component-export.json.gz
```

Reran `CHECK DATABASE` which showed the issue resolved.

Started the nexus service and we are back in business.

Running for a day and a half now and still no reports on any issues so far.

Most important thing now is to never let it run out of disk space again.