How to undo Asset Soft-Delete in NXRM3? - Sonatype Nexus Repository - Sonatype Community

How to undo Asset Soft-Delete in NXRM3?

post by Edwin on Jul 25, 2018

As part of a test with NEXUS3 as nuget package repo, we ran into the problem of vanishing assets. For our users it is unclear if this is a system or a user error, and even though there’s a gut feeling, there is no proof that it wasn’t the system.

Apparently, restoring a soft-deleted asset in NEXUS 2 was as simple as moving it out of the .trash folder and back into the repo tree. What is the equivalent of this in NEXUS 3?

post by jstephens on Jul 26, 2018

Hi Edwin,

Sorry to hear you’re having some unexplained problems. It’d be helpful for me to understand exactly what you mean by “vanishing assets” - Does this mean that you can no longer see the asset in the browse UI / search screens or is it that the NuGet client cannot see it (i.e. it is missing from the NuGet feed for some reason)?

What version of NXRM3 are you running?

In NXRM3 the data is stored in two places; the binaries are stored on disk in the blob-store and then a reference to those binaries is stored in our internal database in the asset table. So there are some possibilities here:

  1. The blob has been marked as soft-deleted but the asset record is still available in the database - In this case you should still be able to see the asset in the browse UI but you will get a warning in the nexus logs when you try to fetch the binary. Each blob on disk has a corresponding .properties file which can contain a deleted=true flag. Removing this flag in these circumstances would restore the asset to you.
  2. The blob has been marked as soft-deleted and the asset record has also been deleted - This is more of a problem because at the moment you would have to remove the deleted flag (as above) but also manually rebuild the asset record. We have added a task to do this automatically for Npm, Yum and Maven but haven’t yet implemented it for NuGet.
  3. The blob is not soft-deleted and the asset record hasn’t been removed but the nupkg is not showing up in the NuGet feed - In this case I would file a bug ticket on our JIRA https://issues.sonatype.org with which packages have a problem and include a support zip.

In the first two possibilities you can get extra information by finding the blob .properties file for a removed asset and find the property “deletedReason”.

post by Edwin on Jul 26, 2018

Hi Joe,

I’m running NEXUS 3:

|version|3.10.0-04| |edition|OSS| |buildRevision|f7b56bf088e8e313a210acbccc8b0739f446b962| |buildTimestamp|2018-04-03-1355-42621|

Multiple packages were uploaded to our nexus hosted Nuget repo. These packages were older versions of packages already in the repo. A couple of days later many of these packages were gone from the UI. I did find these packages in the filesystem with their properties file stating deleted=true and deletedReason=Deleting asset AttachedEntityId{asset->#19:26}

I can not find a log entry telling me who (or what) deleted these packages.

I am not able to restore these packages, aside from having them completely reuploaded.

I did try removing the deletion entries from the properties files and a reindex, but as you said this was futile. How would I rebuild the asset record?

Kind regards,

Edwin

post by mprescott on Jul 27, 2018

Edwin, when you say, “older versions of packages already in the repo,” are you saying that you republished older versions of components that had the exact same coordinates? Name, namespace and version?

If so, only one of those can exist at any one time in a given npm [edit: I mean NuGet, though npm has the same restriction] repository. It’s like overwriting a new file with an older copy; the previous (newer) version is deleted, because they share the same spot.

Is this what’s happening?

post by Edwin on Jul 27, 2018

What is happening is: 2. The blob has been marked as soft-deleted and the asset record has also been deleted

By older version, I mean for example:

  1. Foo-4.8.1.nupkg uploaded July 11
  2. Foo-3.5.2.nupkg uploaded July 13 (this one vanished)
  3. Foo-3.5.2.nupkg reuploaded July 24

Asset 1 is still there.

Asset 2 vanished from the UI and now has “deleted=true” in its .properties file.

Asset 3 is the package that was reuploaded after a new hire couldn’t get Foo-3.5.2.nupkg from the repository.

All systems keep the packages in their cache for efficiency, which is why the missing packages weren’t discovered sooner.

Another user might have deleted these “older version” assets unknowingly, but I can’t find this in the log, and I am unable to undo such deletion.

A limited amount of users had deletion rights, until we find a root cause this has been revoked.

post by mprescott on Jul 27, 2018

That’s clear, thanks—and it’s what I was concerned about.

There can be only one NuGet package called ‘Foo’ of version 3.5.2 in a single repository at any given time. Every time you publish something new to (“Foo”, “3.5.2”), you’re overwriting whatever’s already there, just as if you were overwriting a file. You can’t have two files with the same path and file name on the same file system; you can’t have two packages with the same coordinate in the same repository. For NuGet, the “coordinate” is packageId and version.

Although many ecosystems allow it, we highly recommend against republishing binaries with reused coordinates. If at all possible, adjust your development practices to prevent this from happening.

When you visualize the whole pipeline from development, publication, to downstream caching (possibly in multiple business units, or in a tier of load-balanced read proxies), it can be impossible to know if consumers are referencing the “right one,” or (as happened to you) to tell if the wrong one is still lurking around.

Also, it means your historical record is lossy, since you’re not keeping all of the versions you may care about (for audit reasons, etc.)

In this specific case, you might want to hand-edit the July 13 package to have a unique version number. NuGet allows pre-release qualifiers, so you could have these two packages:

Foo-3.5.2.nupkg

Foo-3.5.2-weForgotToSignItOrWhatever.nupkg

Then you can use the UI Upload feature to upload the ‘old’ version to the repository, where it can live alongside the other one because it has a different coordinate.

post by Edwin on Jul 27, 2018

Actually, this is the desired situation:

We only re uploaded the package after it was already gone from the web UI and “deleted=true” (soft-deleted).

post by Edwin on Jul 30, 2018

So the question remains:

  1. How to find out how (read: by whom) the assets got deleted?
  2. How to restore the soft-deleted NuGet assets in NXRM3?

post by leoviniciuscs on Sep 9, 2019

Hi everyone! Sorry for reviving an 1 year old topic…

@Edwin, did you manage to restore the soft-deleted assets? I’m facing a similar problem and I didn’t find how to do it anywhere.

post by rseddon on Sep 9, 2019

Disclaimer… the following instructions use an implementation detail of the current blob store. As such, they are not officially supported, and they won’t work if the implementation changes.

Associated with each blob in the blob store there is a properties file. So for instance, a particular blob file will consist of two files:

fac5e64d-831b-4b58-98c9-3dd0e4bd1b59.bytes
fac5e64d-831b-4b58-98c9-3dd0e4bd1b59.properties

Within the properties files you’ll see content like this:

#2018-08-22 08:33:33,965-0500
#Wed Aug 22 08:33:33 CDT 2018
@BlobStore.created-by=admin
size=40
@Bucket.repo-name=maven-releases
creationTime=1534944813965
@BlobStore.created-by-ip=127.0.0.1
@BlobStore.content-type=text/plain
@BlobStore.blob-name=org/foo/bar/project/1.2.3/project-1.2.3.jar
sha1=30d7edff3bc3f5e601be9529460b7bd573b1a8b7
deleted=true

The “@Bucket.repo-name” property shows which repository the blob is associated with. The “deleted=true” indicates the blob is in soft deleted state.

To restore the soft deleted file, locate the properties file and remove the “deleted=true” line. Then schedule and run a “repair: reconcile component database from blobstore” task.

There are some limitations associated with the reconcile task, see here for details:

https://help.sonatype.com/display/NXRM3/System+Configuration