Docker push to hosted registry fails with “Missing upload with uuid"

post by kph on Mar 26, 2023

Hello!

I’m asking help about a peculiar issue I’ve not found many instances of on the Internets.

I’ve set up a hosted Docker registry with Nexus 3.49.0-02 running inside the official container. The container is reverse-proxied with traefik:v2.9.9. I’ve also set up an additional subdomain for the docker registries as such:

# Nexus dashboard
- traefik.http.routers.nexus-dash.rule=Host(`nexus.example.com`)
- traefik.http.routers.nexus-dash.service=nexus-dash-service
- traefik.http.routers.nexus-dash.entrypoints=websecure
- traefik.http.routers.nexus-dash.tls=true
- traefik.http.routers.nexus-dash.tls.certresolver=lets-encrypt
- traefik.http.services.nexus-dash-service.loadbalancer.server.port=8081

# Docker registry - group
- traefik.http.routers.nexus-docker-group.rule=Host(`docker-registry.example.com`) && Method(`GET`, `HEAD`)
- traefik.http.routers.nexus-docker-group.service=nexus-docker-group-service
- traefik.http.routers.nexus-docker-group.entrypoints=websecure
- traefik.http.routers.nexus-docker-group.tls=true
- traefik.http.routers.nexus-docker-group.tls.certresolver=lets-encrypt
- traefik.http.services.nexus-docker-group-service.loadbalancer.server.port=30400

# Docker registry - Myapp
- traefik.http.routers.nexus-docker-myapp-1.rule=Host(`docker-registry.example.com`) && Method(`PUT`, `DELETE`, `POST`, `PATCH`)
- traefik.http.routers.nexus-docker-myapp-1.service=nexus-docker-myapp-1-service
- traefik.http.routers.nexus-docker-myapp-1.entrypoints=websecure
- traefik.http.routers.nexus-docker-myapp-1.tls=true
- traefik.http.routers.nexus-docker-myapp-1.tls.certresolver=lets-encrypt
- traefik.http.services.nexus-docker-myapp-1-service.loadbalancer.server.port=30401

Logging in to the registry succeeds, but pushing an image results in the push failing and a layer retrying multiple times until failure.

06dfeb72c4a0: Retrying in 5 seconds
3fdaa70290f5: Waiting
058399f405a5: Waiting
7cd52847ad77: Waiting

^^ Here only one layer is pushing because I’ve limited uploads to 1.

Nexus’ logs show the following stacktrace for each retry and each layer.

2023-03-26 12:15:49,188+0000 WARN  [qtp2104688969-515]  admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PATCH /v2/whoami/blobs/uploads/5bd916ac-c48e-4422-8547-395fa2f0275c
java.lang.IllegalStateException: Missing upload with uuid: 5bd916ac-c48e-4422-8547-395fa2f0275c

I noticed there was a similar bug reported with NEXUS-20640, but that affected versions 3.15.2 and 3.16.1, and was fixed in 3.25.0, which was, if my math is correct, several versions ago. Additionally, the suggested fix, non-parallel uploads, did not work for me.

Also tried pushing from the same vm the Nexus is hosted on but the same error occurs.

I would be very grateful if anyone has ideas on what to check or what to test. I will gladly provide any further details regarding my setup, should it be necessary.

Thanking in advance,

KPH

post by tim-tk on Jan 5, 2024

Hi @kph

we have the same problem with our Nexus Pro 3.62.0. We narrowed it down that this only happens when the associated blob store uses one of our S3 buckets. If we assign a “File” blob store, the problem disappears. Can you confirm that your problem was (is?) related to this?

Regards

Tim