Looking for help, Nexus crashed and now I get error when trying to start

post by sonatype1 on May 8, 2019

matt smith
sonatype1

Hi I just got asked to help with an issue thats was reported that nexus isn’t working, getting 502 bad gateway when trying to load the page, so took a look and can see errors in the logs on startup

below is what I believe to be the relevant snippet from the log file.

Nexus is version nexus-3.12.0-01 according to the logs and the system is running on Ubuntu 16.04LTS

If anyone can shed any light on a possible fix it would be greatly appreciated.

I’ve copied and pasted the log entry below if it best for me to attach it as a separate file please just let me know

2019-05-08 16:35:36,727+0100 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start RESTORE
2019-05-08 16:35:37,526+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Storage ‘accesslog’ was not closed properly. Will try to recover from write ahead log…
2019-05-08 16:35:37,526+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Looking for last checkpoint…
2019-05-08 16:35:37,530+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Found FUZZY checkpoint.
2019-05-08 16:35:37,531+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Data restore procedure from FUZZY checkpoint is started.
2019-05-08 16:35:37,539+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=139, position=791309}} will be skipped during data restore
2019-05-08 16:35:37,540+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} 1 operations were processed, current LSN is LSN{segment=139, position=791309} last LSN is LSN{segment=139, position=791356}
2019-05-08 16:35:37,549+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=139, position=791316}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null, previousCheckpoint=LSN{segment=139, position=791269}} will be skipped during data restore
2019-05-08 16:35:37,550+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=139, position=791356}} will be skipped during data restore
2019-05-08 16:35:37,636+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Storage data recover was completed
2019-05-08 16:35:37,712+0100 INFO [ForkJoinPool.commonPool-worker-2] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/analytics’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
2019-05-08 16:35:37,729+0100 INFO [ForkJoinPool.commonPool-worker-5] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/audit’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
2019-05-08 16:35:37,944+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/accesslog’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
2019-05-08 16:35:38,300+0100 INFO [ForkJoinPool.commonPool-worker-1] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/security’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
2019-05-08 16:35:38,503+0100 WARN [FelixStartLevel] *SYSTEM ROOT - $ANSI{green {db=config}} Storage ‘config’ was not closed properly. Will try to recover from write ahead log…
2019-05-08 16:35:38,504+0100 INFO [FelixStartLevel] *SYSTEM ROOT - $ANSI{green {db=config}} Looking for last checkpoint…
2019-05-08 16:35:40,389+0100 ERROR [FelixStartLevel] *SYSTEM ROOT - Exception `401240DC` in storage `plocal:/net/shares/sonatype-work/nexus3/db/config`: 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
java.lang.OutOfMemoryError: Java heap space
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageV2.getRecord(OWALPageV2.java:112)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OLogSegmentV2.readRecord(OLogSegmentV2.java:748)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.read(ODiskWriteAheadLog.java:848)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.restoreFromWAL(OAbstractPaginatedStorage.java:4673)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.recoverIfNeeded(OAbstractPaginatedStorage.java:3931)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:288)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:143)
at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56)
at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104)
at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44)
at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:252)
at org.sonatype.nexus.orient.DatabaseManagerSupport$$Lambda$42/1734484929.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:231)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$Lambda$41/1743728956.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl.doStart(RestoreServiceImpl.java:76)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
2019-05-08 16:35:40,390+0100 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
java.lang.OutOfMemoryError: Java heap space
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageV2.getRecord(OWALPageV2.java:112)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OLogSegmentV2.readRecord(OLogSegmentV2.java:748)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.read(ODiskWriteAheadLog.java:848)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.restoreFromWAL(OAbstractPaginatedStorage.java:4673)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.recoverIfNeeded(OAbstractPaginatedStorage.java:3931)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:288)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:143)
at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56)
at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104)
at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44)
at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:252)
at org.sonatype.nexus.orient.DatabaseManagerSupport$$Lambda$42/1734484929.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:231)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$Lambda$41/1743728956.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl.doStart(RestoreServiceImpl.java:76)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
2019-05-08 16:35:40,394+0100 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.restore.RestoreServiceImpl - Failed transition: NEW → STARTED
java.lang.reflect.InvocationTargetException: null
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:45)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$EnhancerByGuice$$5bd4efec.start()
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageV2.getRecord(OWALPageV2.java:112)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OLogSegmentV2.readRecord(OLogSegmentV2.java:748)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.read(ODiskWriteAheadLog.java:848)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.restoreFromWAL(OAbstractPaginatedStorage.java:4673)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.recoverIfNeeded(OAbstractPaginatedStorage.java:3931)
at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:288)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)
at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:143)
at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56)
at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104)
at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44)
at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:252)
at org.sonatype.nexus.orient.DatabaseManagerSupport$$Lambda$42/1734484929.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:231)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$Lambda$41/1743728956.accept(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
at org.sonatype.nexus.orient.restore.RestoreServiceImpl.doStart(RestoreServiceImpl.java:76)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
2019-05-08 16:35:41,718+0100 WARN [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Storage ‘component’ was not closed properly. Will try to recover from write ahead log…
2019-05-08 16:35:41,718+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Looking for last checkpoint…
2019-05-08 16:35:41,719+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Found FUZZY checkpoint.
2019-05-08 16:35:41,719+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Data restore procedure from FUZZY checkpoint is started.
2019-05-08 16:35:41,719+0100 WARN [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=141, position=5183499}} will be skipped during data restore
2019-05-08 16:35:41,720+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} 1 operations were processed, current LSN is LSN{segment=141, position=5183499} last LSN is LSN{segment=141, position=5183546}
2019-05-08 16:35:41,720+0100 WARN [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=141, position=5183506}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null, previousCheckpoint=LSN{segment=141, position=5183459}} will be skipped during data restore
2019-05-08 16:35:41,720+0100 WARN [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=141, position=5183546}} will be skipped during data restore
2019-05-08 16:35:41,774+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - $ANSI{green {db=component}} Storage data recover was completed
2019-05-08 16:35:44,564+0100 INFO [ForkJoinPool.commonPool-worker-3] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/component’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)

post by bhanupathak on May 9, 2019

It seems like an issue with Java Heap Space, i am not sure but you can check this article if it helps

post by christianbeland on Jan 6, 2021

Were you able to find a solution for this? I’m in the same situation right now.