Rolling Upgrades in High Availability

High availability (HA) environments may upgrade Nexus Repository without downtime (i.e., rolling upgrade, zero-downtime upgrade).

Terminology

• Zero-Downtime

Zero-downtime upgrades in Nexus Repository mean the system remains available during updates, especially in HA clusters, ensuring uninterrupted artifact access. This is achieved through rolling upgrades, allowing users and builds to continue without service disruption.

• Rolling Upgrades

Updating a Nexus Repository by sequentially upgrading nodes in a high availability cluster, minimizing downtime for artifact access.

Rolling upgrades involve stopping and upgrading each node before then upgrading the database schema. While an individual node is stopped, the other active nodes continue to serve traffic, resulting in no downtime for consumers of the service.

• Mixed Mode

Having different versions of Nexus Repository installed on the nodes of a cluster. While running in mixed mode, new features are not available until all the nodes on the cluster as well as the database schema are upgraded.

While upgrading nodes, administrators may see a warning indicating that the cluster is running in mixed mode. Running in mixed mode outside of an active upgrade is not supported.

Limitations for Rolling Upgrades

The following section details some limitations:

• Review the supported HA rolling upgrade paths before upgrading

You may need to upgrade to an intermediate version before going to the latest version. Upgrading Nexus Repository from 3.71.0 to 3.72.0 is the first version where zero-downtime upgrades are available.

• The HA cluster nodes should only run on different versions temporarily.

Running in mixed mode outside of an active upgrade is not supported. Nexus Repository may become unstable and behave in unexpected ways. Complete the upgrade in as short amount of time as possible to minimize the amount of time Nexus Repository nodes are on different versions.

• Enable rolling upgrades on all nodes of the cluster.

All nodes must have rolling upgrades enabled or they return an error when Nexus Repository detects mixed mode.

• New features are not available until the database schema upgrade is complete.

• New nodes cannot join the cluster while the database schema migration is active

When adding nodes during schema migration, these nodes stop with an error code and log the errors.

• Upgrading the database schema is a one-way process.

You cannot roll back once you proceed with finalizing the upgrade by updating the database schema. You will need to restore from a backup to roll back.

• You cannot configure sticky sessions from within the helm chart

Without sticky sessions, users may see UI inconsistencies during the upgrade process due to changes made between versions. Sticky sessions need to be configured on your load balancer or ingress controller.

Prerequisites for Rolling Upgrades

To use rolling upgrades with zero downtime, you must meet the following prerequisite requirements:

• Requires Nexus Repository Pro 3.71.0 or greater

Nexus Repository Pro in an HA environment with a PostgreSQL database using the supported HA Helm chart version 71.0.0 or above.

• Enable zero-downtime upgrades on each node

Restart each instance after enabling the feature on each node before proceeding with the upgrade.

• Configure sticky sessions on your load balancer

NGINX provides documentation on enabling session persistence.

Enable Zero-Downtime Upgrades

Zero-downtime upgrades must be enabled and restarted on every node before upgrading them. The process to enable is different depending on the environment you are running your cluster. Select the appropriate method below:

• Helm Chart

Update the values.yaml with the zero downtime configuration.

zeroDowntimeEnabled: true

• OpenShift

Update the Nexus Repository custom resource definition (NexusRepo)

spec.statefulset.container.env.zeroDowntimeEnabled = true

• Docker and Kubernetes

Update the statefulset.yaml with the zero downtime configuration.

NEXUS_ZERO_DOWNTIME_ENABLED: true

• Manual Environment Variable

Set the environment variable for every node.

NEXUS_ZERO_DOWNTIME_ENABLED=true

• Manual through nexus.properties

Modify the $data-dir/sonatype-work/nexus3/etc/nexus.properties for each node.

nexus.zero.downtime.enabled = true

Rolling Back on Zero-Downtime Upgrades

Zero downtime upgrades each node to the latest Nexus Repository version before updating the database. At any point before finalizing the changes to the database, the cluster may be rolled back to the version of Nexus Repository running previously.

• Mixed Mode

If the upgrade has not been finalized, the cluster is in mixed mode. Bring down the instance(s) with the higher version to move the nodes back to the older versions.

• Finalize not selected

The finalize button appears when all instances have the upgraded version, but the database is not yet upgraded. At this stage, you can start instances with the previous version to put the cluster back into mixed mode.

• Finalize in progress

Once finalized, rolling back is possible by restoring from a backup.

Phase 1 - Upgrade Each Node

One node at a time; take the node offline, upgrade the node, and start the node using the new Nexus Repository version. Upgrade the nodes by replacing the binaries or follow the instructions for Kubernetes environments.

• Nodes are upgrades one at a time.

• The load balancer distributes requests to active nodes.

• PostgreSQL database remains on the old schema.

Phase 2 - Finalize Upgrade

Note
Finalizing the upgrade is only required if there are database schema changes between Nexus Repository versions and a database schema migration is required. If one is not required, the Upgrade Pending banner and Finalize Upgrade button will not appear. You can then skip Phase 2.

When the upgrade requires a database schema migration, administrators with the nx-atlas-all permission see a banner at the bottom of the screen. There is a delay of up to 10 minutes before the banner appears.

1. Select Finalize Upgrade in the Upgrade Pending banner.

This opens a dialogue to proceed with the database schema migration.

2. Select Acknowledge that you have backed up your database.

Select Continue to proceed with database schema migration.

3. The banner shows that an upgrade is in progress; when complete, it displays an Upgrade Complete banner.

Any errors are captured in the audit log.

4. Reload the user interface to use new features.

Finalize Migration with REST APIs

REST API users can finalize their upgrade via the REST APIs listed in the table below.

The process for finalizing your upgrade and migrating your database schema is as follows:

1. Use a GET request to verify that the state is MIGRATION_REQUIRED

2. Use a POST request to start a migration

3. Poll the GET endpoint; it will return one of the following statuses:

4. MIGRATION_SUCCEED - indicates successful database schema migration. Send a DELETE request to clear the status.

5. MIGRATION_FAILED_RETRYABLE - indicates that a precondition for migration failed; consult the accompanying error message to identify and remediate the cause. After remediation, use a POST to try again.

6. MIGRATION_FAILED - indicates that an error occurred, and we do not believe it can be rectified without Support team assistance.

After successful database schema migration, your upgrade is complete.

Supported Rolling Upgrade Paths

We may occasionally have limitations on eligible version upgrade ranges. Attempting to exceed the upgrade windows will result in new nodes failing to start. Follow this section to stay up to date on any required intermediate versions.