Cleanup Policies

Cleanup Policies

Cleanup policies help you automatically manage repository storage by removing old, unused, or outdated components. By defining rules for what to delete and when, you can maintain repository hygiene, reduce storage costs, and ensure compliance with your organization's retention requirements.

Note that cleanup policies and the associated tasks are not configured by default. Define the policies that best suit your requirement.

Cleanup Policies Workflow

Administrators create cleanup policies. The following sequence is a high level description of using cleanup policies:

  1. Administrators create Cleanup Policies depending on their requirements.

  2. Administrators assign one or more Cleanup Policies to the repository configuration.

See Repository Management

  1. Administrators schedule Cleanup Tasks to run regularly. (e.g., once a night)
Admin - Cleanup repositories using their associated policies  
Admin - Cleanup unused asset blobs

Cleanup tasks "soft delete" components by flagging them for removal. Components still consume space but may be recovered when needed.

See Tasks

  1. Run the compact blob store task on off-peak hours for each blob store to reclaim disk space. These tasks completely removed the components from the storage disk, freeing up the available space.
Admin - Compact blob store

Security Requirements

Only users with admin privileges may create cleanup policies. (i.e., nexus:*). Users with edit repository privileges may add a clean policy to the repository.

nexus:repository-admin:{format}:{repository}:edit

Users with either of the following privileges may modify cleanup tasks:

nx-tasks-update, nx-tasks-all

See Repository Admin

Creating Cleanup Policies

Cleanup Policies are located in the repository section of the settings menu and require admin privileges to create or modify. They are intended for use in one more many repositories typically associated with a specific repository format type.

  1. Navigate to Settings → Repository→ Cleanup Policies.

  2. Select Create Cleanup Policy.

  3. Provide a unique name for the policy

Only letters, digits, underscores(_), hyphens(-), and dots(.) are allowed and may not start with underscore or dot.

  1. Under Format field, select a target format for the policy

The 'All formats' option may be selected for any repository type. However most formats have specific cleanup criteria. See Cleanup Criteria by Format.

  1. Enter a brief description about the policy in the Description field (limit of 400 characters).

  2. New in 3.95 Under Repositories field, add the Available Repositories to Applied Repositories.

  3. Under Cleanup Criteria, select at least one criterion to remove the components. For more details on the available criteria, see Cleanup Criteria.

  4. If you wish to retain the components by the number of versions, select the Number of Versions checkbox and enter the component age. For more details, see Retain Select Versions.

  5. Select Save.

Strategies for Creating a Cleanup Policy

Adding A Cleanup Policy to a Repository

Cleanup policies may be assigned to both proxy and hosted repositories. Users require repository admin edit privileges to edit or add repositories to cleanup policies.

  1. Select the repository from Repositories view in the Settings menu or create a new one

  2. Navigate to the Cleanup section

  3. Optionally use the search filter to limit the available cleanup policies from the list

  4. Select the required cleanup policies from the available section

  5. Use the right-facing arrow to move the cleanup policy to the applied section

  6. Repeat for as many policies that are required

  7. Select the Save button

Previewing Cleanup Policy Results

Before running a cleanup policy on a production repository, preview the results to see which components would be deleted. Previewing is not an audit as the results may differ from the actual run. Take the following steps to preview the results:

  1. Navigate to Settings → Repository→ Cleanup Policies.

  2. Select an existing cleanup policy or create a new policy with your desired criteria.

  3. Under Preview Cleanup Policy Results, select the repository to preview.

Note that there is a 1-minute timeout on the preview to reduce the impact to performance.

PostgreSQL Cleanup Preview

Generate the list of components that the policy would remove as configured. This is a point-in-time audit as results may differ when testing the policy at a different time.

  1. Navigate to Settings → Repository→ Cleanup Policies.

  2. Select an existing cleanup policy or create a new policy with your desired criteria.

  3. Under Preview Cleanup Policy Results, select the repository to preview.

  4. Select Generate CSV Report. The CSV file will be downloaded once the query is complete.

Filename: <cleanup policy name>-<repository name>-<timestamp>.csv  
Fields:   namespace, name, version, path

Generating the cleanup preview CSV takes time depending on deployment size and configuration. The table below provides the approximate estimated generation time:

Deployed on an AWS ECS c6i.4xlarge instance with Aurora PostgreSQL db.r6g.large database  
   ----------------------------------------------------------------------------  
Components in Repository (millions) || Time to Generate CSV Report (minutes)  
    1M  =   ~1min  
    5M  =   ~2min  
20M  =   ~8min  
25M  =  ~15min  
27M  =  ~25min  
30M  =  ~40min

Under Nexus One UI, you can immediately see the list of components marked for cleanup.

Preview Limitations

The preview is a snapshot in time. Components matching the criteria may change if new components are published or existing components are downloaded before the cleanup runs. Preview counts are approximate. The actual number of deleted components may differ slightly due to concurrent changes.

Cleanup Tasks

Nexus Repository automatically creates a few system tasks to soft delete components identified for cleanup by the cleanup policies. These tasks are not manually created and are re-added on a restart of the service when deleted.

This task soft deletes components based on the repository's configured cleanup policies. These tasks may be rescheduled or manually executed. By default, this task is set to run once an hour.

These tasks soft delete orphaned assets that are no longer needed after a component is removed. They are added when a new format has been added as a repository. By default, these tasks run every 30 minutes.

Hard Deleting Components with the Compact Blob Store Task

Cleanup Policies soft delete components by updating their metadata with a marker for deletion. This optimizes the cleanup tasks and allows administrators to recover components when they are accidentally deleted. While Nexus Repository no longer displays soft deleted components in the user interface, they are not immediately deleted from storage and still use storage space.

Use the Admin - Compact blob store task to reclaim disk space and permanently delete the components from storage.

Sonatype Nexus Repository Cloud

The compact blob store task is managed and scheduled automatically. It runs daily and deletes blobs that have been soft-deleted for at least 3 days. After blobs are deleted, they may be retained in underlying storage for up to 45 days before permanent removal.

Create and schedule the Admin - Compact blob store task to reclaim disk space. Add this task for every blob store that requires cleanup.

Retain Soft-Deleted Blobs for a Period of Time Before Hard Deleting

When configuring the Admin - Compact blob store task you may set the Blobs Older Than property to temporarily retain deleted artifacts for a designated number of days before deleting them from storage. The default value is 0 days.

Interaction with Azure Blob Store Cleanup

The compact blob store task requests the Azure blob store to mark blobs for deletion. These are later hard-deleted during garbage collection on the Azure side. This varies on whether the soft delete feature is enabled as described in Azure's documentation.

Interaction with AWS S3

Before release 3.80.0, the Admin - Compact blob store task did not delete components from AWS S3 blob stores. As of release 3.80.0, AWS S3 blob stores use the "Blobs Older Than" property for files in the S3 bucket. The S3 blob store configuration with the Expiration Days fields is no longer used for the S3 lifecycle policy.

Docker Cleanup Strategies

Docker's tagging, manifests, and layers are unique ways of managing components and assets that require additional configuration when designing a cleanup strategy.

See Components and Assets in Docker to learn more about the docker format.

  1. Docker - Delete incomplete uploads

Soft-delete uploads to the temporary storage that are not complete

  1. Admin - Cleanup repositories using their associated policies

Soft-delete old published or downloaded docker components i.e. tags, not layers or manifests

  1. Docker - Delete unused manifests and images

Soft-delete orphaned layers and manifests no longer referenced by tags, possibly orphaned by cleanup policies

Warning

Docker manifest requests by digest do not include a reference to a tag, therefore they are cleaned up with this task.

docker pull image_name@DIGEST

Use this task with caution when cleaning up images that are not tagged.

Once the above tasks have run, the following tasks are needed to hard delete the components and reclaim space depending on your deployment.

Additional Information

Use the Component Usage (Days) criterion to clean up components that have never been downloaded. This criterion removes components that haven't been downloaded in a specified number of days. The date the component was published is used when the component has never been downloaded.

Content Replication does not replicate the deletion of components on remote repositories. Cleanup policies only remove components from the specific instance on which it is run. The remote repository requires its own setup of cleanup policies.

Nexus Repository instances using PostgreSQL databases use SQL-based cleanup by default.

See the metrics at Cleanup Performance Data.

Use the following Support article to determine the space your repositories are using.

Investigating Blob Store and Repository Size and Space Usage

Maven - Delete unused SNAPSHOT  
Repository - Delete unused components

Cleanup Criteria

Cleanup Criteria by Format Table

The table below lists the available cleanup criteria and the formats to which they apply:

Format Component Age Component Usage Release Type Retain Select Versions Asset Name Matcher
#### All Formats
#### APT
#### Bower²
#### CocoaPods
#### Conan
#### Conda
#### Docker¹
#### GitLFS
#### Go
#### Helm
#### Maven
#### npm
#### NuGet
#### p2
Pub
#### PyPI
#### R
#### Raw
#### RubyGems
Swift
Terraform
#### Yum

¹ - Cleanup only evaluates tagged manifests for Docker.

² - Bower functionality is for proxy repositories only.

Component Age (Days)

This criteria sets how long to keep content based on component age.

Component Usage (Days)

This criteria sets how long to keep content based on when a component was last downloaded. The published or updated dates are used when the component has never been downloaded.

Release Type

Use to set the cleanup policy to either PRELEASES or RELEASES. Prereleases are different by format:

Versions contain the -SNAPSHOT phrase

Uses semantic versioning where a version is a prerelease when it contains the dash "-" character

The non-case-sensitive "release" property in the RPM header contains one of the following:

alpha, beta, rc, pre, prerelease, snapshot

Retain Select Versions

Use the Number of Versions criteria to keep only the most recent versions of each component in a repository. This is useful for repositories where components are published frequently and older versions consume unnecessary storage. This feature requires Nexus Repository Pro and a PostgreSQL database.

The following list describes how Nexus Repository normalizes and sorts component versions for each supported format:

NEW IN 3.70

The version number is used, available for the release type Releases.

The age of the manifest is used.

NEW IN 3.95

npm versions follow semantic versioning. Pre-release versions have lower precedence than release versions. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

APT (Debian) versions follow Debian version comparison rules. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Python package versions follow PEP 440 versioning. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Go module versions follow semantic versioning. Pseudo-versions are sorted by timestamp. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Helm chart versions follow semantic versioning. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

NuGet package versions follow NuGet versioning rules. Versions are sorted numerically for major, minor, and patch segments. Pre-release versions have lower precedence than release versions. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Dart pub package versions follow semantic versioning. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

RubyGems versions follow RubyGems versioning rules. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Swift package versions follow semantic versioning. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Terraform provider and module versions follow semantic versioning. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

YUM (RPM) versions follow RPM version comparison rules. Retaining select versions is available for both Releases and Pre-Releases/Sanpshots.

Asset Name Matcher

Rules are based on the component name, namespace, or path in the repository. Supported regular expression patterns differ between the legacy OrientDB and the newer PostgreSQL and H2 environments.

When migrating to PostgreSQL or H2, legacy cleanup policies may result in more assets being removed than expected.

PostgreSQL and H2 Expressions

In H2 or PostgreSQL environments, the Asset Name Matcher uses Java regular expressions.

When migrating to PostgreSQL, revise cleanup policies to include the leading slash in asset matcher names. Failure to do so may result in assets not being matched and cleaned up as expected.

OrientDB Expressions

Expressions in OrientDB uses the Elastic Search regular expression query syntax, from Apache Lucene.

Comparison between OrientDB and PostgreSQL

This example contains the following assets for consideration:

Pattern
 antlr.*

Repository
 /antlr/antlr/2.7.2/antlr-2.7.2.jar
 /org/antlr/antlr-master/3.1.3/antlr-master-3.1.3.pom

Expression Examples

The following examples demonstrate a specific asset name matcher against a repository and the remaining components after using the matcher in a cleanup policy.

/hello/-/hello-0.0.[1-2].tgz

Repository
/hello/-/hello-0.0.1.tgz
/hello/-/hello-0.0.2.tgz
/hello/-/hello-0.0.3.tgz

Remaining
/hello/-/hello-0.0.3.tgz

/(org|com)/.*

Repository
/org/example/test.jar
/com/example/test.jar
/test/example/test.jar

Remaining
/test/example/test.jar

/org/sonatype/^(team2)/.*

Repository
/org/sonatype/team1/ui/5.0/ui-5.0.jar
/org/sonatype/team2/format/1.0/format-1.0.jar
/org/sonatype/team3/database/10.0/database-10.0.jar

Remaining
/org/sonatype/team2/format/1.0/format-1.0.jar

/pool/main/z/zsh/zsh-common_5.4.2-3ubuntu3_all.deb

Repository
/pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb
/pool/main/z/zsh/zsh_5.4.2-3ubuntu3_amd64.deb
/pool/main/z/zsh/zsh-common_5.4.2-3ubuntu3_all.deb

Remaining
/pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb
/pool/main/z/zsh/zsh_5.4.2-3ubuntu3_amd64.deb

For the expression com/sonatype

^(?!.*?com/sonatype/).*$

Repository
/org/apache/commons/lang3/StringUtils.java
/com/google/guava/collect/ImmutableList.java
/java/util/ArrayList.java
/com/sonatype/nexus/repository/Repository.java
/org/eclipse/aether/impl/internal/com/sonatype/Logging.java

Remaining
/com/sonatype/nexus/repository/Repository.java
/org/eclipse/aether/impl/internal/com/sonatype/Logging.java

^(?!.*?(?:/org/sonatype/|/com/sonatype/)).*$

Repository
/org/apache/maven/project/MavenProject.java
/com/google/inject/AbstractModule.java
/my/own/sonatype/unrelated/file.txt
/home/user/.m2/repository/com/sonatype/nexus/nexus-client-core/2.7/nexus-client-core-2.7.jar
/org/sonatype/suffix/file.xml
/another/path/prefix/com/sonatype/suffix/data.json

Remaining
/home/user/.m2/repository/com/sonatype/nexus/nexus-client-core/2.7/nexus-client-core-2.7.jar
/org/sonatype/suffix/file.xml
/another/path/prefix/com/sonatype/suffix/data.json

Search results

No results found