Recovery Mode

Recovery Mode is a special operational state that ensures the system is aware of a reconciliation process in the database and blob storage after system outages or data inconsistencies. It ensures that data repair operations run safely by preventing the execution of tasks that may affect data recovery efforts.

Recover Mode feature is available only for users with admin privileges and is available only in self-hosted instances.

Enabling Recovery Mode

Prerequisites

To use recovery mode, you must meet the following prerequisites:

• You must have admin privileges
• The deployment must be self-hosted
• Consult with support team before enabling

How to Enable

Recovery Mode can be enabled through either the Management Page or via API.

Option A: Management Page

1. Navigate to Support > Recovery Mode.
2. Click Enable Recovery Mode.
3. Confirm success notification that appears.

Option B: API
Endpoint

POST /v1/recovery-mode

Administrative credentials are required for authorization.

When Recovery Mode is enabled, the following actions occur:

• Conflicting background tasks are automatically canceled, including Asset Blob Cleanup, Compact Blob Store, Repository Move, Blob Store Group Member Removal.
• The Recovery Mode flag is set.
• A warning banner is displayed to administrators indicating that Recovery Mode is active.

Using Recovery Mode for reconciliation

After enabling Recovery Mode, run the required data repair tasks to reconcile inconsistencies between the database and blob storage:

• Run the Repair - Data Repair Plan task.
• Run the Repair - Execute Data Repair Plan task.

For detailed information about these tasks, see Data Repair Tasks.

Verify repair completion

To verify reconciliation results, use the reconciliation reports as described in Data Repair Tasks. For more information, see Accessing Reconciliation Reports.

Exiting Recovery Mode

Prerequisites

• You must have admin privileges
• Both reconciliation tasks must be stopped or done

How to Exit

Recovery Mode can be exited through either the Management Page or via API.

Option A: Management Page

1. Navigate to Support > Recovery Mode.
2. Verify that no reconciliation tasks are running.
3. Click Disable Recovery Mode.
4. If unexecuted reconciliation plans exist, confirm via modal dialog.
5. Confirm success notification that appears.

Note

The Disable button will remain unavailable if reconciliation tasks are still running.

Option B: API
Endpoint

DELETE /v1/recovery-mode

Administrative credentials are required for authorization.

When Recovery Mode is exited, the following actions occur:

• Recovery mode flag is removed.
• Admin banner is hidden.
• Conflicting tasks are unblocked.
• Unexecuted reconciliation plans remain (can be deleted via API if desired).

Recovery Mode Behavior

Banner Notification

When Recovery Mode is enabled, a banner is displayed at the top of the screen, consistent with the styling of zero-downtime upgrade notifications.

• The banner includes a direct link to the Recovery Mode Management page.
• Visibility of the banner is restricted to administrative users only.

Restricted Tasks During Recovery Mode

While Recovery Mode is active, only these following specific tasks are blocked:

• Asset Blob Cleanup Task
• Cleanup Task
• Compact Blob Store Task
• Docker GC Task
• Docker GC Custom Task
• Repository Move Task
• Blob Store Group Member Removal Task

If any of these tasks are initiated during Recovery Mode, they will fail immediately. A log entry will be generated indicating that Recovery Mode is enabled and that reconciliation must be completed before the task can proceed.

All other tasks, including reconciliation-related tasks, will continue to operate as normal.

Best Practices

• Consult Support - Always verify Recovery Mode is appropriate for your situation.
• Backup First - Perform a full database backup before executing reconciliation plans.
• Execute Plans - Complete reconciliation before disabling Recovery Mode to avoid stale plans.
• Plan Downtime - Reconciliation duration depends on blob store size and system scale.

Troubleshooting

Cannot Disable Recovery Mode

• Reconciliation tasks are still running, so you can either wait for them to complete or manually stop them, noting that stopping them may leave the reconciliation process incomplete.

Conflicting Task Fails

• Recovery Mode is enabled, which is expected behavior, and it can be disabled once reconciliation is complete.

Banner Not Visible

• Verify you have admin privileges and using self-hosted deployment.
• Refresh browser or check console for errors.

Unexecuted Plans Remain

• Plans are not automatically deleted if they are not executed. Any unexecuted plan will remain in the system until it is explicitly removed.
• Plans in the PLANNED or EXECUTE states can be deleted using one of the following APIs:

Delete a specific plan

DELETE /v1/api/plan/{planId}

Delete all plans in PLANNED or EXECUTE state

DELETE /v1/api/plan