Questions regarding Reconcile component database from blob store

post by renatsakenov on Sep 20, 2021

Hello everyone,

We are using Nexus Repository Manager OSS, 3.30.1-01.

Recently, around 1 - 1.5 TB of artifacts of a hosted maven repository have been accidentally removed (via cleanup and compact blob store). The blobstore is of type file.

To restore, I copied to missing artifacts (blobs / files on the filesystem) from a backup to the corresponding blobstore.

Afterwards I ran reconcile component database from blob store. This was about a month after the deletion.

I chose to not restore the database.

My assumption was that Nexus would pick up the restored artifacts on the filesystem and update the database, so that, in the end, all missing artifacts are restored and available again.

This approach is working so far, but there are a few questions / problems:

The total amount of deleted blobs are approx. 1.1 million (according to the display on the admin panel → Blob Stores → Blob count)

• When running the reconcile job, the amount of blobs increased (certainly due to the job running, not by other uploads) but the total size did not change. Why is that?
• The reconcile job ran for 48 hours without interruption and according to the display in the admin panel, approx. 11.000 blobs were added. So apparently the job would need to run for approx. 200 days straight to restore all blobs. Is this plausible?

When looking at the logs, there are approx. 900.000 lines that contain “Restored asset, blob store:” and when checking a few blob-ids manually, they all seem to be unique.

How do these numbers relate to each other?

• I canceled the reconcile task and did not run the search / browse repair task since I’m not sure how long those tasks will run and how this will affect availability. Will the display on the admin panel update when running those tasks?
• When I canceled the reconcile task, the following log line was printed:

*SYSTEM org.sonatype.nexus.blobstore.restore.orient.OrientRestoreMetadataTask - ---- Elapsed time: 2.005 d, processed: 1961883, un-deleted: 0 ----

I assume this is not directly related to the reconcile task, since it says RestoreMetadataTask, or does it? What exactly does the value of processed mean in this context?

I could not find documentation which describes the mentioned tasks in detail, I hope someone can answer my questions and especially clarify the mixed readings that I got.

Thank you and Best regards

Renat

post by renatsakenov on Sep 28, 2021

Update:

The job finished after an effective runtime of 61 hours. (around 85 hours in total, because of the interruption. After the interrupted to took about 24 hours until the task got the point where it broke off the last time)

While everything should now be restored and some missing artifacts are visible again, I don’t think all of them reappeared. The overview under the admin panel → blob stores also has outdated information.

post by atharva.inamdar on Oct 12, 2022

still facing an issue in v3.41.1.