Looking for help, Nexus crashed and now I get error when trying to start

post by sonatype1 on May 8, 2019

Hi I just got asked to help with an issue thats was reported that nexus isn’t working, getting 502 bad gateway when trying to load the page, so took a look and can see errors in the logs on start up

below is what I believe to be the relevant snippet from the log file.

Nexus is version nexus-3.12.0-01 according to the logs and the system is running on Ubuntu 16.04 LTS

If anyone can shed any light on a possible fix it would be greatly appertained.

I’ve copied and pasted the log entry below if it best for me to attach it as a separate file please just let me know

2019-05-08 16:35:36,727+0100 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start RESTORE 2019-05-08 16:35:37,526+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Storage ‘accesslog’ was not closed properly. Will try to recover from write ahead log… 2019-05-08 16:35:37,526+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Looking for last checkpoint… 2019-05-08 16:35:37,530+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Found FUZZY checkpoint. 2019-05-08 16:35:37,531+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Data restore procedure from FUZZY checkpoint is started. 2019-05-08 16:35:37,539+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=139, position=791309}} will be skipped during data restore 2019-05-08 16:35:37,540+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} 1 operations were processed, current LSN is LSN{segment=139, position=791309} last LSN is LSN{segment=139, position=791356} 2019-05-08 16:35:37,549+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=139, position=791316}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null, previousCheckpoint=LSN{segment=139, position=791269}} will be skipped during data restore 2019-05-08 16:35:37,550+0100 WARN [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=139, position=791356}} will be skipped during data restore 2019-05-08 16:35:37,636+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - $ANSI{green {db=accesslog}} Storage data recover was completed 2019-05-08 16:35:37,712+0100 INFO [ForkJoinPool.commonPool-worker-2] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/analytics’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) 2019-05-08 16:35:37,729+0100 INFO [ForkJoinPool.commonPool-worker-5] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/audit’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) 2019-05-08 16:35:37,944+0100 INFO [ForkJoinPool.commonPool-worker-4] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/accesslog’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) 2019-05-08 16:35:38,300+0100 INFO [ForkJoinPool.commonPool-worker-1] *SYSTEM ROOT - Storage ‘plocal:/net/shares/sonatype-work/nexus3/db/security’ is opened under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) 2019-05-08 16:35:38,503+0100 WARN [FelixStartLevel] *SYSTEM ROOT - $ANSI{green {db=config}} Storage ‘config’ was not closed properly. Will try to recover from write ahead log… 2019-05-08 16:35:38,504+0100 INFO [FelixStartLevel] *SYSTEM ROOT - $ANSI{green {db=config}} Looking for last checkpoint… 2019-05-08 16:35:40,389+0100 ERROR [FelixStartLevel] *SYSTEM ROOT - Exception 401240DC in storage plocal:/net/shares/sonatype-work/nexus3/db/config: 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)

java.lang.OutOfMemoryError: Java heap space

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageV2.getRecord(OWALPageV2.java:112)

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OLogSegmentV2.readRecord(OLogSegmentV2.java:748)

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.read(ODiskWriteAheadLog.java:848)

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OAbstractPaginatedStorage.restoreFromWAL(OAbstractPaginatedStorage.java:4673)

at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.recoverIfNeeded(OAbstractPaginatedStorage.java:3931)

at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:288)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:259)

at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:143)

at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56)

at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104)

at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44)

at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:252)

at org.sonatype.nexus.orient.DatabaseManagerSupport$$Lambda$42/1734484929.apply(Unknown Source)

at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:231)

at org.sonatype.nexus.orient.restore.RestoreServiceImpl$$Lambda$41/1743728956.accept(Unknown Source)

at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)

at java.util.stream.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)

at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)

at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)

at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.doExec:289)

at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.doInvoke(401);

at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.doInvoke(734);

at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160);

at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:174);

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233);

at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418);

at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583);

at org.sonatype.nexus.orient.restore.RestoreServiceImpl.doStart(RestoreServiceImpl.java:76);

at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67);

2019-05-08 16:35:40,390+0100 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed

java.lang.OutOfMemoryError: Java heap space

at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageV2.getRecord(OWALPageV2.java:112);

post by bhanupathak on May 9, 2019

It seems like an issue with Java Heap Space, i am not sure but you can check this article if it helps

post by christianbeland on Jan 6, 2021

Were you able to find a solution for this? I’m in the same situation right now.