Using Nexus as a proxy registry for a disconnected OpenShift install

post by vincent.cojot on Aug 26, 2024

Hi all,

I’m attempting to use Nexus as a local proxy registry for deploying OCP clusters.

I’ve configured upstream docker proxy repos to quay.io, registry.redhat.io and a few others.

Then, I’ve aggregated those repos (each with their pull secrets) under a group hosted on Nexus on port 5000.

This works fine for image-based downloads.

Upstream:

podman pull quay.io/openshift-release-dev/ocp-release:4.16.5-x86_64

My registry:

podman pull registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-release:4.16.5-x86_64

Both of these work without issues.

However, if I try to deploy Openshift using Nexus as my registry, I get errors for the sha256 images (digests?), e.g:

This one works (using upstream):

podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842

but this one fails:

 podman pull registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842

and it reports: “manifest unknown”

In the nexus outbound Log, I can then see 401 errors such as these:

[26/Aug/2024:12:36:09 +0000] 7742700|uhc-1L9AybTZWMqYYhqpfGHL945CLJ0 "GET https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 HTTP/1.1" 401 112 299 "Nexus/3.71.0-06 (OSS; Linux; 4.18.0-553.16.1.el8_10.x86_64; amd64; 17.0.12)" [qtp565611798-4260]
[26/Aug/2024:12:36:11 +0000] 7742700|uhc-1L9AybTZWMqYYhqpfGHL945CLJ0 "GET https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/manifests/sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 HTTP/1.1" 401 112 303 "Nexus/3.71.0-06 (OSS; Linux; 4.18.0-553.16.1.el8_10.x86_64; amd64; 17.0.12)" [qtp565611798-4260]

Is this doable? Am I doing something wrong?

For reference, this is what I am using in my install-config:

imageContentSources:
- mirrors:
  - registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-v4.0-art-dev
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - registry.lasthome.solace.krynn:5000/openshift-release-dev/ocp-release
  source: quay.io/openshift-release-dev/ocp-release

post by vincent.cojot on Aug 26, 2024

I’ve reached out to OpenShift support and they said that the only way this could fail with a 401 on quay.io is if the requests aren’t properly authenticated:

The v2/auth endpoint on Quay will only return a 401 if you have bad credentials. There would be no other reason to do so. Without credentials:

# podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842...
Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842: reading manifest sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842 in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized

With credentials:

# podman pull --authfile pull-secret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e4b8d40d25ba75162a201b45ff09dea4dd9fb13d078bbd6dd09276266df6842...
Getting image source signatures
Copying blob 8deeda9a2ea3 done   |
Copying blob 68957586c3c2 done   |
Copying blob ca1636478fe5 done   |
Copying blob 312897d9e4ab done   |
Copying config e1f53e55a0 done   |
Writing manifest to image destination
e1f53e55a022852fd7f988ce0a92f262adf73de64a1ed99700040f8df0f098ae

post by mpiggott on Sep 3, 2024

I think this sounds like a potential bug that should be posted to Issues · sonatype/nexus-public · GitHub

post by vincent.cojot on Dec 16, 2024

Got it working with Nexus 3.75.1 and a small patch to kcli.

Details here: small patch for disconnected OCP deploy with regular registries (like Nexus) · Issue #757 · karmab/kcli · GitHub