NXRM 3 restarts after many broken pipe warnings

post by mr.boutobza on Jun 14, 2021

I am encountring for sometime a problem which is: that nxrm3 (ver 3.29.2) restarts after many broken pipe warnings, this is an example of the content of nexus.log just before nxrm3 reboots

the problem is realy annoying as is causing my nxrm3 to reboot (downtime) many times (approximately 10 times/week)

    <a lot of broken pipe warnings similar to the mentionned one just below>

2021-06-13 06:33:24,680+0200 WARN  [qtp198125444-7659]  admin org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/reponame/v2/myapp/serviceA/blobs/sha256:968013a6f0d1ceec8448f0e822bb4e2dc050d391329577791bd0e3367
    org.eclipse.jetty.io.EofException: null
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)
        at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)
        at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:833)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:112)
        at org.sonatype.nexus.repository.view.Payload.copy(Payload.java:61)
        at org.sonatype.nexus.repository.view.Content.copy(Content.java:116)
        at org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender.send(DefaultHttpResponseSender.java:81)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:229)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.doFilter:61)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.executeChain:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilterInternal:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.doFilter:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.executeChain:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilterInternal:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.doFilter:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.executeChain:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.doFilterInternal:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.executeChain:450)
        at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.executeChain:96)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.invoke:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.doCall:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.call:83)
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.execute:387)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.doFilterInternal:362)
        at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.doFilterInternal:112)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.doFilter:125)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.doFilter:80)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.doFilter:116)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.doFilter:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.doFilter:79)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.doFilter:101)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.doFilter:98)
        at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.dispatch:104)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.doFilter:133)
        at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.doFilter:73)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.doFilter:201)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler$Chain.doFilter:1609)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.doHandle:561)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.handle:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.nextHandle:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.doHandle:1612)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.doScope:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.doScope:1582)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.doHandle:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.doScope:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.doScope:1582)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.doHandle:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.doScope:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.doScope:1582)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.doHandle:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.nextScope:188)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at org.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at com.codahale.metrics.jetty9.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.doFilter:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.doFilter:73)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.doFilter:201)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler$Chain.doFilter:1609)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler$Chain.doHandle:561)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.handle:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at com.codahale.metrics.jetty9.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.doFilter:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.doFilter:82)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.doFilter:201)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler$Chain.doFilter:1609)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler$Chain.doHandle:561)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.handle:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.handle:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.handle:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.handle:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.handle:146)
    ```

some points that may be important to debug:

- the errors (broken pipe) shows only when pulling a docker image with “docker pull”  
- docker client is installed on redhat 7.4 (docker is redhat version see below)

```bash
docker/1.13.1 go/go1.10.3 kernel/3.10.0-1127.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)
    ```
- nxrm3 use `s3` as blobstores
- orientdb is stored on network device (NFSv3)
- CPU => `7`, RAM=> `25G`
- `INSTALL4J_ADD_VM_PARAMS`=> `-Xms4G -Xmx4G -XX:MaxDirectMemorySize=6717M -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap`

any help to resolve/debug this issue will be appreciated

## post by mpiggott on Jun 15, 2021

The errors are unlikely to be the cause of of Nexus restarting, though they indicate probably you have a reverse proxy prematurely terminating requests.

## post by mr.boutobza on Jun 15, 2021

i am using traefik as my reverse proxy i set it’s configurations to be as in [Run Behind a Reverse Proxy](https://help.sonatype.com/repomanager3/installation/run-behind-a-reverse-proxy#RunBehindaReverseProxy-nginx),

please note also that mentioned warnings doesn’t happen with every docker pull command !  
and can you please suggest what can cause nexus to restart ? because i can’t find any unusual logs in nexus.log or request.log files or can you share some steps to debug the situation.

thank you in advance

## post by arjen on Jul 2, 2021

[@mr.boutobza](https://community.sonatype.com/u/mr.boutobza) - Did you find the cause of this, or even better a solution?

I updated our server from 1.22 to 1.31.0 and have had several restarts overnight with loads of eofexceptions in the logs, but I can’t replicate any issues myself.