# Cleanup Policies Cleanup policies help you automatically manage repository storage by removing old, unused, or outdated components. By defining rules for what to delete and when, you can maintain repository hygiene, reduce storage costs, and ensure compliance with your organization's retention requirements. Note that cleanup policies and the associated tasks are not configured by default. Define the policies that best suit your requirement. ## Cleanup Policies Workflow Administrators create cleanup policies. The following sequence is a high level description of using cleanup policies: 1. Administrators create `Cleanup Policies` depending on their requirements. 2. Administrators assign one or more `Cleanup Policies` to the repository configuration. See [Repository Management](https://help.sonatype.com/en/repository-management.html "Repository Management") 3. Administrators schedule `Cleanup Tasks` to run regularly. (e.g., once a night) ``` Admin - Cleanup repositories using their associated policies Admin - Cleanup unused asset blobs ``` Cleanup tasks "soft delete" components by flagging them for removal. Components still consume space but may be recovered when needed. See [Tasks](https://help.sonatype.com/en/tasks.html "Tasks") 4. Run the compact blob store task on off-peak hours for each blob store to reclaim disk space. These tasks completely removed the components from the storage disk, freeing up the available space. ``` Admin - Compact blob store ``` ## Security Requirements Only users with admin privileges may create cleanup policies. (i.e., `nexus:*`). Users with edit repository privileges may add a clean policy to the repository. ``` nexus:repository-admin:{format}:{repository}:edit ``` Users with either of the following privileges may modify cleanup tasks: ``` nx-tasks-update, nx-tasks-all ``` See [Repository Admin](https://help.sonatype.com/en/privileges.html#UUID-31a8c800-61e6-4a12-68c8-5ff90fbcf139_bridgehead-idm234684513948045 "Repository Admin") ## Creating Cleanup Policies Cleanup Policies are located in the repository section of the settings menu and require admin privileges to create or modify. They are intended for use in one more many repositories typically associated with a specific repository format type. 1. Navigate to _Settings_ → _Repository_→ _Cleanup Policies_. 2. Select _Create Cleanup Policy_. 3. Provide a unique name for the policy Only letters, digits, underscores(_), hyphens(-), and dots(.) are allowed and may not start with underscore or dot. 4. Under _Format_ field, select a target format for the policy The '`All formats`' option may be selected for any repository type. However most formats have specific cleanup criteria. See [Cleanup Criteria by Format](https://help.sonatype.com/en/cleanup-policies.html#cleanup-criteria-by-format-table "Cleanup Criteria by Format Table"). 5. Enter a brief description about the policy in the _Description_ field (_limit of 400 characters_). 6. New in 3.95 Under _Repositories_ field, add the _Available Repositories_ to _Applied Repositories_. 7. Under _Cleanup Criteria_, select at least one criterion to remove the components. For more details on the available criteria, see [Cleanup Criteria](https://help.sonatype.com/en/cleanup-policies.html#cleanup-criteria "Cleanup Criteria"). 8. If you wish to retain the components by the number of versions, select the _Number of Versions_ checkbox and enter the component age. For more details, see [Retain Select Versions](https://help.sonatype.com/en/cleanup-policies.html#retain-select-versions "Retain Select Versions"). 9. Select _Save_. ### Strategies for Creating a Cleanup Policy - Cleanup Policies are intended to target components to be removed from either hosted or proxy repositories. - The criteria of the policy are combined to remove only the components that meet every condition specified. - Multiple cleanup policies may be applied to the same repository. - Policies can have overlapping criteria targeting the same components. ## Adding A Cleanup Policy to a Repository Cleanup policies may be assigned to both proxy and hosted repositories. Users require repository admin edit privileges to edit or add repositories to cleanup policies. 1. Select the repository from Repositories view in the Settings menu or create a new one 2. Navigate to the Cleanup section 3. Optionally use the search filter to limit the available cleanup policies from the list 4. Select the required cleanup policies from the available section 5. Use the right-facing arrow to move the cleanup policy to the applied section 6. Repeat for as many policies that are required 7. Select the `Save` button ## Previewing Cleanup Policy Results Before running a cleanup policy on a production repository, preview the results to see which components would be deleted. Previewing is not an audit as the results may differ from the actual run. Take the following steps to preview the results: 1. Navigate to _Settings_ → _Repository_→ _Cleanup Policies_. 2. Select an existing cleanup policy or create a new policy with your desired criteria. 3. Under _Preview Cleanup Policy Results_, select the repository to preview. Note that there is a 1-minute timeout on the preview to reduce the impact to performance. ### PostgreSQL Cleanup Preview Generate the list of components that the policy would remove as configured. This is a point-in-time audit as results may differ when testing the policy at a different time. 1. Navigate to _Settings_ → _Repository_→ _Cleanup Policies_. 2. Select an existing cleanup policy or create a new policy with your desired criteria. 3. Under _Preview Cleanup Policy Results_, select the repository to preview. 4. Select _Generate CSV Report_. The CSV file will be downloaded once the query is complete. ``` Filename: --.csv Fields: namespace, name, version, path ``` Generating the cleanup preview CSV takes time depending on deployment size and configuration. The table below provides the approximate estimated generation time: ``` Deployed on an AWS ECS c6i.4xlarge instance with Aurora PostgreSQL db.r6g.large database ---------------------------------------------------------------------------- Components in Repository (millions) || Time to Generate CSV Report (minutes) 1M = ~1min 5M = ~2min 20M = ~8min 25M = ~15min 27M = ~25min 30M = ~40min ``` Under Nexus One UI, you can immediately see the list of components marked for cleanup. ### Preview Limitations The preview is a snapshot in time. Components matching the criteria may change if new components are published or existing components are downloaded before the cleanup runs. Preview counts are approximate. The actual number of deleted components may differ slightly due to concurrent changes. ## Cleanup Tasks Nexus Repository automatically creates a few system tasks to soft delete components identified for cleanup by the cleanup policies. These tasks are not manually created and are re-added on a restart of the service when deleted. - #### Cleanup service: Admin - Cleanup repositories using their associated policies This task soft deletes components based on the repository's configured cleanup policies. These tasks may be rescheduled or manually executed. By default, this task is set to run once an hour. - #### Cleanup unused {format} blobs from nexus: Admin - Cleanup unused asset blobs These tasks soft delete orphaned assets that are no longer needed after a component is removed. They are added when a new format has been added as a repository. By default, these tasks run every 30 minutes. ## Hard Deleting Components with the Compact Blob Store Task Cleanup Policies soft delete components by updating their metadata with a marker for deletion. This optimizes the cleanup tasks and allows administrators to recover components when they are accidentally deleted. While Nexus Repository no longer displays soft deleted components in the user interface, they are not immediately deleted from storage and still use storage space. Use the `Admin - Compact blob store` task to reclaim disk space and permanently delete the components from storage. **Sonatype Nexus Repository Cloud** The compact blob store task is managed and scheduled automatically. It runs daily and deletes blobs that have been soft-deleted for at least 3 days. After blobs are deleted, they may be retained in underlying storage for up to 45 days before permanent removal. Create and schedule the `Admin - Compact blob store` task to reclaim disk space. Add this task for every blob store that requires cleanup. ### Retain Soft-Deleted Blobs for a Period of Time Before Hard Deleting When configuring the `Admin - Compact blob store` task you may set the `Blobs Older Than` property to temporarily retain deleted artifacts for a designated number of days before deleting them from storage. The default value is 0 days. ### Interaction with Azure Blob Store Cleanup The compact blob store task requests the Azure blob store to mark blobs for deletion. These are later hard-deleted during garbage collection on the Azure side. This varies on whether the soft delete feature is enabled as described in [Azure's documentation](https://learn.microsoft.com/en-us/rest/api/storageservices/delete-blob). ### Interaction with AWS S3 Before release 3.80.0, the `Admin - Compact blob store` task did not delete components from AWS S3 blob stores. As of release 3.80.0, AWS S3 blob stores use the "`Blobs Older Than`" property for files in the S3 bucket. The S3 blob store configuration with the `Expiration Days` fields is no longer used for the S3 lifecycle policy. ## Docker Cleanup Strategies Docker's tagging, manifests, and layers are unique ways of managing components and assets that require additional configuration when designing a cleanup strategy. See [Components and Assets in Docker](https://help.sonatype.com/en/repository-manager-concepts.html#docker-components "Docker Components") to learn more about the docker format. 1. #### Docker - Delete incomplete uploads Soft-delete uploads to the temporary storage that are not complete 2. #### Admin - Cleanup repositories using their associated policies Soft-delete old published or downloaded docker components i.e. tags, not layers or manifests 3. #### Docker - Delete unused manifests and images Soft-delete orphaned layers and manifests no longer referenced by tags, possibly orphaned by cleanup policies **Warning** Docker manifest requests by digest do not include a reference to a tag, therefore they are cleaned up with this task. ``` docker pull image_name@DIGEST ``` Use this task with caution when cleaning up images that are not tagged. Once the above tasks have run, the following tasks are needed to hard delete the components and reclaim space depending on your deployment. - Run the `Admin - Compact blob store` task for file-based blob stores ## Additional Information - #### Clean Up Components That Have Never Been Downloaded Use the `Component Usage (Days)` criterion to clean up components that have never been downloaded. This criterion removes components that haven't been downloaded in a specified number of days. The date the component was published is used when the component has never been downloaded. - #### Cleanup Policies Does Not Remove Components from Replicated Repositories Content Replication does not replicate the deletion of components on remote repositories. Cleanup policies only remove components from the specific instance on which it is run. The remote repository requires its own setup of cleanup policies. - #### SQL-Based Cleanup Performance Nexus Repository instances using PostgreSQL databases use SQL-based cleanup by default. See the metrics at [Cleanup Performance Data](https://help.sonatype.com/en/cleanup-performance-data.html "Cleanup Performance Data"). - #### Determining the Space Your Repositories Are Using Use the following Support article to determine the space your repositories are using. [Investigating Blob Store and Repository Size and Space Usage](https://support.sonatype.com/hc/en-us/articles/115009519847) - #### Replace the following tasks with Cleanup Policies ``` Maven - Delete unused SNAPSHOT Repository - Delete unused components ``` ## Cleanup Criteria ### [Cleanup Criteria by Format Table](https://help.sonatype.com/en/cleanup-policies.html#cleanup-criteria-by-format-table) The table below lists the available cleanup criteria and the formats to which they apply: | Format | Component Age | Component Usage | Release Type | Retain Select Versions | Asset Name Matcher | | :-- | :-- | :-- | :-- | :-- | :-- | | #### All Formats | | | | | | | #### APT | | | | | | | #### Bower² | | | | | | | #### CocoaPods | | | | | | | #### Conan | | | | | | | #### Conda | | | | | | | #### Docker¹ | | | | | | | #### GitLFS | | | | | | | #### Go | | | | | | | #### Helm | | | | | | | #### Maven | | | | | | | #### npm | | | | | | | #### NuGet | | | | | | | #### p2 | | | | | | | **Pub** | | | | | | | #### PyPI | | | | | | | #### R | | | | | | | #### Raw | | | | | | | #### RubyGems | | | | | | | **Swift** | | | | | | | **Terraform** | | | | | | | #### Yum | | | | | | ¹ \- Cleanup only evaluates tagged manifests for Docker. ² \- Bower functionality is for proxy repositories only. ### Component Age (Days) This criteria sets how long to keep content based on component age. - Proxy repositories: based on when the component was first downloaded - Hosted repositories: based on when the component was uploaded or updated ### Component Usage (Days) This criteria sets how long to keep content based on when a component was last downloaded. The published or updated dates are used when the component has never been downloaded. ### Release Type Use to set the cleanup policy to either `PRELEASES` or `RELEASES`. Prereleases are different by format: - #### Maven Versions contain the `-SNAPSHOT` phrase - #### npm Uses semantic versioning where a version is a prerelease when it contains the dash "`-`" character - #### Yum The non-case-sensitive "`release`" property in the RPM header contains one of the following: ``` alpha, beta, rc, pre, prerelease, snapshot ``` ### Retain Select Versions Use the _Number of Versions_ criteria to keep only the most recent versions of each component in a repository. This is useful for repositories where components are published frequently and older versions consume unnecessary storage. This feature requires Nexus Repository Pro and a PostgreSQL database. The following list describes how Nexus Repository normalizes and sorts component versions for each supported format: NEW IN 3.70 - **Maven** The version number is used, available for the release type `Releases`. - **Docker** The age of the manifest is used. NEW IN 3.95 - **npm** npm versions follow semantic versioning. Pre-release versions have lower precedence than release versions. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **APT** APT (Debian) versions follow Debian version comparison rules. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **PyPI** Python package versions follow PEP 440 versioning. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **Go** Go module versions follow semantic versioning. Pseudo-versions are sorted by timestamp. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **Helm** Helm chart versions follow semantic versioning. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **NuGet** NuGet package versions follow NuGet versioning rules. Versions are sorted numerically for major, minor, and patch segments. Pre-release versions have lower precedence than release versions. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **Pub** Dart pub package versions follow semantic versioning. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **RubyGems** RubyGems versions follow RubyGems versioning rules. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **Swift** Swift package versions follow semantic versioning. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **Terraform** Terraform provider and module versions follow semantic versioning. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. - **YUM** YUM (RPM) versions follow RPM version comparison rules. Retaining select versions is available for both `Releases` and `Pre-Releases/Sanpshots`. ### Asset Name Matcher Rules are based on the component name, namespace, or path in the repository. Supported regular expression patterns differ between the legacy OrientDB and the newer PostgreSQL and H2 environments. When migrating to PostgreSQL or H2, legacy cleanup policies may result in more assets being removed than expected. #### PostgreSQL and H2 Expressions In H2 or PostgreSQL environments, the Asset Name Matcher uses Java regular expressions. - Not compatible with OrientDB Lucene regular expressions - Java regular expressions may match any part of the component path When migrating to PostgreSQL, revise cleanup policies to include the leading slash in asset matcher names. Failure to do so may result in assets not being matched and cleaned up as expected. #### OrientDB Expressions Expressions in OrientDB uses the [Elastic Search regular expression](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-regexp-query.html#regexp-syntax) query syntax, from Apache Lucene. - Not compatible with Perl (PCRE) or Java `util.regex.Pattern` regular expressions - Expressions must match the entire name when wildcards are not used. - Asset names do not require a leading slash and use a limited set of operators - Asset matchers in OrientDB are different than the asset request path value used when evaluating content selector or routing rule expressions #### Comparison between OrientDB and PostgreSQL This example contains the following assets for consideration: ``` Pattern antlr.* Repository /antlr/antlr/2.7.2/antlr-2.7.2.jar /org/antlr/antlr-master/3.1.3/antlr-master-3.1.3.pom ``` - **OrientDB** \- the first component is matched while the second is not - **H2 or PostgreSQL** \- both components are matched #### Expression Examples The following examples demonstrate a specific asset name matcher against a repository and the remaining components after using the matcher in a cleanup policy. - #### Components in a version range ``` /hello/-/hello-0.0.[1-2].tgz Repository /hello/-/hello-0.0.1.tgz /hello/-/hello-0.0.2.tgz /hello/-/hello-0.0.3.tgz Remaining /hello/-/hello-0.0.3.tgz ``` - #### Components with a specific path ``` /(org|com)/.* Repository /org/example/test.jar /com/example/test.jar /test/example/test.jar Remaining /test/example/test.jar ``` - #### Cleanup components matching a specific team ``` /org/sonatype/^(team2)/.* Repository /org/sonatype/team1/ui/5.0/ui-5.0.jar /org/sonatype/team2/format/1.0/format-1.0.jar /org/sonatype/team3/database/10.0/database-10.0.jar Remaining /org/sonatype/team2/format/1.0/format-1.0.jar ``` - #### A specific component ``` /pool/main/z/zsh/zsh-common_5.4.2-3ubuntu3_all.deb Repository /pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb /pool/main/z/zsh/zsh_5.4.2-3ubuntu3_amd64.deb /pool/main/z/zsh/zsh-common_5.4.2-3ubuntu3_all.deb Remaining /pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb /pool/main/z/zsh/zsh_5.4.2-3ubuntu3_amd64.deb ``` - #### Cleanup components that DO NOT MATCH this expression For the expression `com/sonatype` ``` ^(?!.*?com/sonatype/).*$ Repository /org/apache/commons/lang3/StringUtils.java /com/google/guava/collect/ImmutableList.java /java/util/ArrayList.java /com/sonatype/nexus/repository/Repository.java /org/eclipse/aether/impl/internal/com/sonatype/Logging.java Remaining /com/sonatype/nexus/repository/Repository.java /org/eclipse/aether/impl/internal/com/sonatype/Logging.java ``` - #### Cleanup components that DO NOT MATCH more than one expression ``` ^(?!.*?(?:/org/sonatype/|/com/sonatype/)).*$ Repository /org/apache/maven/project/MavenProject.java /com/google/inject/AbstractModule.java /my/own/sonatype/unrelated/file.txt /home/user/.m2/repository/com/sonatype/nexus/nexus-client-core/2.7/nexus-client-core-2.7.jar /org/sonatype/suffix/file.xml /another/path/prefix/com/sonatype/suffix/data.json Remaining /home/user/.m2/repository/com/sonatype/nexus/nexus-client-core/2.7/nexus-client-core-2.7.jar /org/sonatype/suffix/file.xml /another/path/prefix/com/sonatype/suffix/data.json ``` ## Search results No results found