# High Availability in Amazon Web Services

High Availability should be deployed and tested in a development environment before deploying to production. This reference architecture is designed to protect against the following scenarios:

- An AWS Availability Zone (AZ) outage within a single AWS region
- A node/server (i.e., EC2) failure
- A Nexus Repository service failure

Use this architecture for the following use cases:

- Nexus Repository Pro user looking for a deployment option to reduce downtime.
- Require automatic failover and fault tolerance in your deployment requirement.
- An Elastic Kubernetes Service (EKS) cluster is in your deployment pipeline.
- Nexus Repository is using an external PostgreSQL database.

## Requirements

- Meet the [System Requirements](https://help.sonatype.com/en/sonatype-nexus-repository-system-requirements.html "Sonatype Nexus Repository System Requirements")
- Meet the [System Requirements for High Availability Deployments](https://help.sonatype.com/en/system-requirements-for-high-availability-deployments.html "Requirements for High Availability")
- Adjust the [PostgreSQL Max\_Connections](https://help.sonatype.com/en/postgresql-max-connections.html "Adjust the PostgreSQL Max Connections")
- AWS account with permissions to access the services described below.

### Limitations

1. The ability to upgrade with zero downtime (rolling upgrades) is only available from release 3.71.0 forward; 3.72.0 is the first release you can upgrade to without shutting down all active nodes.
   
   See [HA upgrade documentation](https://help.sonatype.com/en/upgrading-nexus-repository-in-an-ha-environment.html "Upgrading Nexus Repository in a Kubernetes Environment")
2. We do not recommend deploying HA clusters across regions due to the large number of database interactions involved in HA.
3. In the absence of a distributed solution for logging, AWS EBS volumes are required to persist the log files that are needed for support zip creation.

## Setting Up the Infrastructure

**Note**  
Unless otherwise specified, all steps detailed below are still required if you are planning to use the HA/Resiliency Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)).

### Step 1 - AWS EKS Cluster

Nexus Repository runs on an AWS EKS cluster spread across two or more AZs within a single AWS region. You can control the number of nodes by setting the min, max, and desired nodes parameters. EKS ensures that the desired number of Nexus Repository instances runs within the cluster. If something causes an instance or the node to go down, AWS will spin up another one. If an AZ becomes unavailable, AWS spins up new node(s) in the secondary AZ.

Begin by setting up the EKS cluster in the AWS web console, ensuring your nodes are spread across two or more AZs in one region. The EKS cluster must be created in a virtual private cloud (VPC) with two or more public or private subnets in different AZs. AWS provides instructions for managed nodes (i.e., EC2) [in their documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html).

#### [Step 1 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-1-validation--optional--161967_body)

**Note**  
This validation section describes AWS products, field names, and UI elements as documented at the time of writing. Note that AWS may change labels or UI elements over time, and these validation steps may not be updated immediately.

1. Validate the VPCs, subnets, and their network connections in the AWS VPC console under _VPC → Your VPCs → <Your VPC Name>_. Consider [AWS's best practices for VPCs and subnets](https://aws.github.io/aws-eks-best-practices/networking/subnets/#:~:text=Please%20refer%20to%20the%20EKS,public%20or%20a%20private%20subnet.):
2. In the AWS IAM console, validate that your nodegroup has an IAM role that has the following policies assigned (See [AWS IAM Roles documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)):
   - AmazonEKSWorkerNodePolicy
   - AmazonEC2ContainerRegistryReadOnly
   - AmazonEKS\_CNI\_Policy
   - AmazonS3FullAccess (for S3 access)

If you want to use CloudWatch to stream logs, add the below policies
   - CloudWatchFullAccess
   - CloudWatchAgentServerPolicy (to use AmazonCloudWatchAgent on servers)

3. In the AWS IAM console under _IAM → Roles → <Your EKS Cluster Role>_, ensure that the EKS cluster's role has `AmazonEKSClusterPolicy` permissions listed in the _Permissions_tab.
4. Ensure that the role above is attached to the EKS cluster by checking the _Cluster IAM role ARN_ section under _EKS_ → _Clusters_ → _<cluster-name>_in the AWS IAM console.
5. Ensure that the cluster security group allows traffic into the cluster under the _Inbound rules_ tab located in the _VPC → Security Groups_ section of the AWS console.
6. Each Sonatype Nexus Repository instance requires at least 4 CPUs and 8GB memory. Ensure the EC2 nodes for the nodegroup are configured appropriately.
7. Ensure that there are a sufficient number of IP addresses available in the two subnets to account for the cluster, nodes, and other Kubernetes resources.
8. Ensure that the EKS API server endpoint is set to your desired setting: Public, Private, or both. This decision should be based on your requirements for exposing the EKS cluster outside of the VPC. Follow [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) to control access to EKS clusters.
9. Enable DNS resolution and DNS hostnames for the VPC in the _DNS settings_ section.
10. Under _Security → Network ACLs → <Your-NACL>,_ ensure that the Network ACL's inbound and outbound rules allow traffic into and from the subnets. (See the [AWS NACL documentation](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html).)
11. Ensure IAM OIDC identity provider for your cluster exists by using the following command:
   ```bash
   aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4
   ```
   If it does not exist, use the command below to create an IAM OIDC identity provider:
   ```bash
   eksctl utils associate-iam-oidc-provider --cluster <cluster-name>  --approve
   ```
   If eksctl is not installed, create the IAM OIDC provider using the AWS web console from _IAM → Identity providers_.
12. In the AWS EKS console under _EKS → Clusters → <Cluster-Name> → Add-ons,_ check that you have installed the add-ons below and that you are using versions that are compatible with your EKS version:
 - [Amazon VPC CNI](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html)  
 - [CoreDNS](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)  
 - [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html)  
 - [Amazon EBS CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html)

13. In the AWS CLI, run the following command; it should return the cluster information.
   ```bash
   aws eks describe-cluster --name <cluster-name>  --region <region-code>
   ```
14. Run the kubectl command below to ensure that nodes are created:
   ```bash
   kubectl get nodes
   ```
15. In the AWS console under _EC2_ → _Auto Scaling_ → _Auto Scaling groups_, ensure that the auto scaling group exists for the EKS cluster and has the desired number of nodes and AZs; this will have been created automatically with the EKS cluster. See the [AWS auto scaling groups documentation](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html).

### Step 2 - Elastic File System Dynamic Provisioning

In this step, create an Elastic File System (EFS) and a service account with an IAM policy to allow the cluster to use EFS storage. Install the EFS CSI driver to create access points for accessing EFS. EFS storage is required to persist files required by the Nexus Repository pods when Kubernetes attempts to restart a pod in a different availability zone.

1. Follow the AWS documentation to create an EFS in the AWS console.

- The EFS CSI driver creates an access endpoint when provisioning storage.
   - The VPC used by you EFS must be the same as the one accessing your EKS cluster.
   - Set the security group to allow incoming traffic on `port 2049` from your EKS cluster security group. This is the security group used by all nodes in your EKS cluster.

See [Create a file system using the AWS CLI](https://docs.aws.amazon.com/efs/latest/ug/creating-using-create-fs.html#creating-using-fs-part1-console)

2. Create an IAM policy using the AWS-provided example and saving it as `iam_policy.json`

See [Github file: iam-policy-example.json](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/iam-policy-example.json)

- Use the following command to create the policy after saving your file:
     ```bash
     aws iam create-policy \
       --policy-name EFSCSIControllerIAMPolicy \
       --policy-document file://iam-policy.json
     ```
   - Take note of the policy URL as you will need it in the next step.

3. Create an IAM service account using a command like the following:
   ```bash
   eksctl create iamserviceaccount \
       --cluster=<cluster-name> \
       --region=<region-name> \
       --namespace=kube-system \
       --name=efs-csi-controller-sa \
       --override-existing-serviceaccounts \
       --attach-policy-arn=arn:aws:iam::<aws-account-number>:policy/EFSCSIControllerIAMPolicy \
       --approve
   ```

4. Install the EFS CSI driver via helm charts using the following command. Update the parameters to match your deployment.
   ```bash
   helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/

helm upgrade -i aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver \
       --namespace kube-system \
       --set image.repository=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver \
       --set controller.serviceAccount.create=false \
       --set controller.serviceAccount.name=efs-csi-controller-sa
   ```

See [AWS documentation on installing the EFS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html)

5. Review the required updates to the `values.yaml` before installing the Nexus Repository helm chart as documented in the README file.
   See [Helm Chart README file](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/README.md)

### Step 3 - Service Account

This is not required when using the HA Helm chart as this step is automatic.

The Kubernetes Service Account is associated with an IAM role containing the policies for access to S3 and AWS Secrets Manager. The `statefulset` uses this account when deploying Nexus Repository containers.

1. Run the [Service Account YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-service-accounts.yaml) to establish the service account

#### [Step 3 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-3-validation--optional--161967_body)

1. Ensure that the IAM role has the `AmazonS3FullAccess` permissions
2. Set a trust relationship between the role and Kubernetes service account.

The trust relationship information is under _IAM → Roles → Trust relationships_ in the AWS console.
3. Verify that the service account is the same one specified for the statefulset in the values.yaml

Learn more from the [AWS documentation on how to use trust policies with IAM roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/)

### Step 4 - AWS Aurora PostgreSQL Cluster

An Aurora PostgreSQL cluster containing three databases (one writer node and two replicas) spread across three AZs in the region where you've deployed your EKS provides an external database for Nexus Repository configurations and component metadata. We recommend creating these nodes in the same AZs as the EKS cluster.

AWS provides instructions on creating an Aurora database cluster [in their documentation](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html).

#### [Step 4 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-4-validation--optional--161967_body)

1. Ensure that the [pg\_trgm (trigram) module](https://www.postgresql.org/docs/current/pgtrgm.html) is installed as a plugin using the command below:
   ```bash
   SELECT * FROM pg_extension;
   ```
2. Verify that the DB security group allows the cluster security group at port 5432.
3. Ensure you have created a [custom DB Cluster Parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithDBClusterParamGroups.html#USER_WorkingWithDBClusterParamGroups.CreatingCluster), adjusted the `max_connections` parameter, and assigned it to the DB cluster. See our [Prerequisite Step: Adjust max\_connections](https://help.sonatype.com/en/postgresql-max-connections.html "Adjust the PostgreSQL Max Connections").
4. Connect to the PostgreSQL database as described in the [AWS documentation for connecting to a DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Connecting.html) to validate that it is accessible with the hostname, ports, and credentials that you configured in the Kubernetes YAML files or Helm chart.
5. Log into PostgreSQL to validate that there is a database called "nexus" (or whatever preferred name you used when completing step 1 of Migrating to PostgreSQL).

### Step 5 - AWS Load Balancer Controller

The AWS Load Balancer Controller allows you to provision an AWS ALB via an Ingress type specified in your Kubernetes deployment YAML file. This load balancer, which is provisioned in one of the public subnets specified when you create the cluster, allows you to reach the Nexus Repository pod from outside the EKS cluster. This is necessary because the nodes on which EKS runs the Nexus Repository pod are in private subnets and are otherwise unreachable from outside the EKS cluster.

Follow the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) to deploy the AWS LBC to your EKS cluster. If you are unfamiliar with the AWS Load Balancer Controller and Ingress, read the [AWS blog on AWS Load Balancer Controller](https://aws.amazon.com/blogs/containers/exposing-kubernetes-applications-part-2-aws-load-balancer-controller/).

If you encounter any errors when creating a load balancer using the AWS Load Balancer Controller, follow the steps in [AWS's troubleshooting knowledgebase article](https://aws.amazon.com/premiumsupport/knowledge-center/load-balancer-troubleshoot-creating/).

#### [Step 5 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-5-validation--optional--161967_body)

1. Navigate to _IAM → Roles → <Load balancer role name> → Trust relationships_ in the AWS console. Ensure that the service account is associated with an IAM role and that there is a trust relationship defined between the IAM role and the service account. Follow the [AWS documentation to validate the trust relationship](https://repost.aws/knowledge-center/eks-load-balancer-webidentityerr). (Also see the [AWS documentation on how to use trust policies with IAM roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/).

2. Run the command below to validate deployment creation:
   ```bash
   kubectl get deployment -n kube-system aws-load-balancer-controller
   ```
   If the deployment for the load balancer will not start, you may need to delete and recreate the load balancer.

3. If your ingress was not successfully created after several minutes, run the following command to view the AWS Load Balancer Controller logs. These logs may contain error message that you can use to diagnose issues with your deployment.
   ```bash
   kubectl logs -f -n kube-system -l app.kubernetes.io/instance=aws-load-balancer-controller
   ```

4. Run the following command and ensure that the service account created has the AWS IAM role annotated (also see the [AWS documentation on configuring role and service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html)):
   ```bash
   kubectl describe sa aws-load-balancer-controller -n kube-system
   ```
   If not annotated, use the following command:
   ```bash
   kubectl annotate serviceaccount -n $namespace   $service_account    eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role
   ```

5. Ensure that the IAM role created has the `AWSLoadBalancerControllerIAMPolicy` permission. (See the [AWS documentation on IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).

6. Validate that the IAM service account is listed in the Cloudformation Stack under _CloudFormation → Stacks_in the AWS CloudFormation console. (See the [AWS documentation on working with stacks](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html).

7. Validate that the ALB created appears under _EC2 → Load balancers_ in the AWS console.

### Step 6 - Kubernetes Namespace

**Tip**
If you plan to use the HA Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)), you do not need to perform the step below; the Helm chart will create this for you.

A namespace allows you to isolate groups of resources in a single cluster. Resource names must be unique within a namespace, but not across namespaces. See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) about namespaces for more information.

To create a namespace, use a command like the one below with the kubectl command-line tool:
   ```bash
   kubectl create namespace <namespace>
   ```

#### [Step 6 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-6-validation--optional--161967_body)

Expand Validation Steps

Use the command below to validate namespace creation:
   ```bash
   kubectl get ns <namespace>
   ```

### Step 7 - Licensing

There are two options to use for license management:

1. **AWS Secrets Manager** - When using the HA/Resiliency Helm Chart or YAML files
2. **Configure License in Helm Chart** - When using the HA/Resiliency Helm Chart but not using AWS Secrets Manager

#### Licensing Option 1 - AWS Secrets Manager

AWS Secrets Manager stores the license and the database connection properties. During a failover, the Secrets Manager retrieves the license when the container starts.

As of Nexus Repository release 3.74.0, you have the option of using an external secrets operator rather than the secret store CSI driver. This is recommended since it supports several external secret stores.

##### External Secrets Operator (Recommended)

To use an external secrets operator, locate the external secrets portion of your values.yaml (shown in the code block below) and complete [the steps detailed in the Helm chart README](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha#injecting-required-secrets-into-your-nexus-repository-pod)
   ```yaml
   externalsecrets:
     enabled: false
     secretstore:
       name: nexus-secret-store
       spec:
         provider:
   #        aws:
   #          service: SecretsManager
   #          region: us-east-1
   #          auth:
   #            jwt:
   #              serviceAccountRef:
   #                name: nexus-repository-deployment-sa # Use the same service account name as specified in serviceAccount.name
   # Example for Azure
   #    spec:
   #      provider:
   #        azurekv:
   #          authType: WorkloadIdentity
   #          vaultUrl: "https://xx-xxxx-xx.vault.azure.net"
   #          serviceAccountRef:
   #            name: nexus-repository-deployment-sa # Use the same service account name as specified in serviceAccount.name
     secrets:
       nexusSecret:
         enabled: false
         refreshInterval: 1h
         providerSecretName: nexus-secret.json
         decodingStrategy: null # For Azure set to Base64
       database:
         refreshInterval: 1h
         valueIsJson: false
         providerSecretName: dbSecretName # The name of the AWS SecretsManager/Azure KeyVault/etc. secret
         dbUserKey: username #  The name of the key in the secret that contains your database username
         dbPasswordKey: password # The name of the key in the secret that contains your database password
         dbHostKey: host # The name of the key in the secret that contains your database host
       admin:
         refreshInterval: 1h
         valueIsJson: false
         providerSecretName: adminSecretName # The name of the AWS SecretsManager/Azure KeyVault/etc. secret that contains your Nexus Repository admin password
         adminPasswordKey: "nexusAdminPassword" # The name of the key in the secret that contains your nexus repository admin password
       license:
         providerSecretName: nexus-repo-license.lic # The name of the AWS SecretsManager/Azure KeyVault/etc. secret that contains your Nexus Repository license
         decodingStrategy: null # Can be Base64
         refreshInterval: 1h
   ```

##### Secret Store CSI Driver Instructions

If you do not wish to use an external secrets operator, you can use the Secrets Store CSI driver. Follow the [AWS documentation for Secrets Store CSI Drivers](https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/README.md) to mount the license secret, which is stored in AWS Secrets Manager, as a volume in the pod running Nexus Repository.
  
**Note**
Read all the instructions below to build a complete set of parameters to include in the commands.

**When you reach the command for installing the Secret Store CSI Driver**, include the `--set syncSecret.enabled=true` flag. This will ensure that secrets are automatically synced from AWS Secrets Manager into the Kubernetes secrets specified in the [secrets YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-secrets.yaml).
Only the AWS CLI supports storing a binary license file. AWS [provides documentation](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/put-secret-value.html) for using a `--secret-binary` argument in the CLI.
The command will look as follows:
   ```bash
   aws secretsmanager create-secret --name supersecretlicense --secret-binary fileb://license-file.lic --region <region>
   ```
This will return a response such as this:
   ```json
   {
      "VersionId": "4cd22597-f0a9-481c-8ccd-683a5210eb2b",
      "Name": "supersecretlicense",
      "ARN": "arn:aws:secretsmanager:<region>:<account id>:secret:license-string"
   }
   ```
You will put the ARN value in the [secrets YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-secrets.yaml).

**Note**
If updating the license (e.g., when renewing your license and receiving a new license binary), you'll need to restart the Nexus Repository pod after uploading the license to the AWS Secrets Manager. The AWS CLI command for updating a secret is `put-secret-value`.

**When you reach the command for creating an IAM service account**, follow these additional instructions:
- You must include two additional command parameters when running the command: `--role-only` and `--namespace <nexusrepo namespace>`
  - It is important to include the `--role-only` option in the `eksctl create iamserviceaccount` command so that the Helm chart manages the Kubernetes service account.
- The namespace you specify to the `eksctl create iamserviceaccount` must be the same namespace into which you will deploy the Nexus Repository pod.
  - Although the namespace does not exist at this point, you must specify it as part of the command. Do not create that namespace manually beforehand; the Helm chart will create and manage it.
  - You should specify this same namespace as the value of `nexusNs` in your values.yaml.

Your command should look similar to the following where `$POLICY_ARN` is the access policy created in a previous step when you follow the [AWS documentation](https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/README.md).
   ```bash
   eksctl create iamserviceaccount --name <sa-name> --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts --role-only
   ```
   Note that the command will not create a Service Account but will create an IAM role only.

##### [AWS Secrets Manager Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#aws-secrets-manager-validation--optional--161967_body)

**Note**
This section assumes you have installed the HA/Resiliency Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)) or [Sample AWS YAMLs](https://github.com/sonatype/nxrm3-ha-repository/tree/main/sample-aws-ha-yamls).

1. Ensure that the nodes in the EKS cluster have an IAM role with `GetSecretValue` and `DescribeSecret` permissions and that this role has a trust relationship with the service account (you can find the trust relationship information under _IAM → Roles → Role Name → Trust relationships_ in the AWS console). (Also see the [AWS documentation on how to use trust policies with IAM roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/).)
2. Run the following command to validate that the IAM role for the Kubernetes service account exists:
   ```bash
   eksctl get iamserviceaccount --cluster "$CLUSTERNAME"  --region "$REGION"
   ```
3. Run the command below to validate that the service account for the EBS controller has the AWS IAM role annotated (also see the [AWS documentation on configuring role and service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html)):
   ```bash
   kubectl describe sa ebs-csi-controller-sa -n kube-system
   ```
   If it is not annotated, use the command below:
   ```bash
   kubectl annotate serviceaccount -n $namespace   $service_account     eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role
   ```
4. Using the command below, validate that the secrets were created when you installed the Helm chart or used the provided YAML file:
   ```bash
    kubectl get secrets -n <namespace>
   ```
   If they are missing, it is possible that the `--syncSecret.enabled=true` flag was not included with the Helm command for [installing the Secret Store CSI Driver](https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/README.md).
   To fix this, uninstall the Helm chart and install it again with the `--syncSecret.enabled=true` flag.

#### Licensing Option 2 - Configure License in Helm Chart

This option is only for those using the HA/Resiliency Helm Chart but **not** AWS Secrets Manager.
   ```yaml
     license:
       name: nexus-repo-license.lic
       licenseSecret:
         enabled: false
         file: # Specify the license file name with --set-file license.licenseSecret.file="file_name" helm option
         fileContentsBase64: your_license_file_contents_in_base_64
         mountPath: /var/nexus-repo-license
   ```
   1. In the `values.yaml`, locate the license section as shown below.
   2. Change the `license.licenseSecret.enabled` to `true`
   3. Do **one** of the following:
   1. Specify your license file with the following:
         ```bash
         --set-file license.licenseSecret.file="file_name"
         ```
   2. Put the base64 representation of your license file in the below value:
         ```bash
         license.licenseSecret.fileContentsBase64
         ```

### Step 8 - AWS CloudWatch (Optional but Recommended)

**Tip**
If you plan to use the HA/Resiliency Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)), you do not need to perform this step; the Helm chart will handle it for you.

When running Nexus Repository on Kubernetes, it is possible for it to run on different nodes in different AZs over the course of the same day. In order to be able to access Nexus Repository's logs from nodes in all AZs, you must externalize your logs. We recommend that you externalize your logs to CloudWatch for this architecture; however, if you choose not to use CloudWatch, you must externalize your logs to another external log aggregator to make sure that the Nexus Repository logs survive node crashes or when pods are scheduled on different nodes as is possible when running Nexus Repository on Kubernetes. Follow [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html) to set up Fluent Bit for gathering logs from your EKS cluster.

When first installed, Nexus Repository sends task logs to separate files. Those files do not exist at startup and only exist as the tasks are being run. In order to facilitate sending logs to CloudWatch, you need the log file to exist when Nexus Repository starts up. The [nxrm-logback-tasklogfile-override.yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-nxrm-logback-tasklogfile-override.yaml) available in our sample files GitHub repository sets this up.

Once Fluent Bit is set up and running on your EKS cluster, apply the [fluent-bit.yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-fluent-bit.yaml) to configure it to stream Nexus Repository's logs to CloudWatch. The specified Fluentbit YAML sends the logs to CloudWatch log streams within a nexus-logs log group.

[AWS also provides documentation](https://docs.aws.amazon.com/cloudwatch/index.html) for setting up and using CloudWatch.

#### [Step 8 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-8-validation--optional--161967_body)

**Note**
This section assumes the following:
- **Helm-Based Deployments** - You must have set the`fluentbit.enabled`to "true" in values.yaml and installed the HA Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)).
- **Non-Helm Deployments** - You must have edited the [fluentbit yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-fluent-bit.yaml) as appropriate for your environment and applied it to your cluster.

1. Ensure that the IAM role assigned to your EKS node(s) has the following permissions:
   - `CloudWatchFullAccess`
   - `CloudWatchAgentServerPolicy`

2. Ensure that the DaemonSet and Pods for the fluent-bit configuration are running using a command like the following:
   ```bash
   kubectl get all -n amazon-cloudwatch
   ```

3. Log into the fluent-bit pod using a command like the following:
   ```bash
   kubectl exec -ti --namespace amazon-cloudwatch <fluent-bit pod name>  -- sh
   ```

4. Validate that the logs are being written to `/var/log/containers`.

### Step 9 - External DNS (Optional)

If you are using or wish to use our [Docker Subdomain Connector](https://help.sonatype.com/en/docker-subdomain-connector.html "Docker Subdomain Connector") feature, you will need to use [external-dns](https://github.com/kubernetes-sigs/external-dns) to create 'A' records in AWS Route 53.

You must meet all Docker Subdomain Connector feature requirements, and you must specify an HTTPS certificate ARN in [Ingress YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-ingress.yaml).

You must also add your Docker subdomains to your values.yaml.

#### Permissions

You must first ensure you have appropriate permissions. To grant these permissions, open a terminal that has connectivity to your EKS cluster and run the following commands:

**Note**
The commands below do not register a domain for you; you must have a registered domain before completing this step.

1. Use the following to create the policy JSON file.
   ```bash
   cat <<'EOF' >> external-dns-r53-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [\
       {\
         "Effect": "Allow",\
         "Action": [\
           "route53:ChangeResourceRecordSets"\
         ],\
         "Resource": [\
           "arn:aws:route53:::hostedzone/*"\
         ]\
       },\
       {\
         "Effect": "Allow",\
         "Action": [\
           "route53:ListHostedZones",\
           "route53:ListResourceRecordSets"\
         ],\
         "Resource": [\
           "*"\
         ]\
       }\
     ]
   }
   EOF
   ```

2. Use the following to set up permissions to allow external DNS to create route 53 records.
   ```bash
   aws iam create-policy --policy-name "AllowExternalDNSUpdates" --policy-document file://external-dns-r53-policy.json
   
   POLICY_ARN=$(aws iam list-policies --query 'Policies[?PolicyName==`AllowExternalDNSUpdates`].Arn' --output text)
   
   EKS_CLUSTER_NAME=<Your EKS Cluster Name>
   
   eksctl utils associate-iam-oidc-provider --cluster $EKS_CLUSTER_NAME --approve
   
   ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
   OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --query "cluster.identity.oidc.issuer" --output text | sed -e 's|^https://||')
   ```

**Note**
   The value you assign to the `EXTERNALDNS_NS` variable below should be the same as the one you specify in your values.yaml for `namespaces.externaldnsNs`.
   ```bash
   EXTERNALDNS_NS=nexus-externaldns
  
   cat <<-EOF > externaldns-trust.json
   {
       "Version": "2012-10-17",
       "Statement": [\
           {\
               "Effect": "Allow",\
               "Principal": {\
                   "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"\
               },\
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {\
                   "StringEquals": {\
                       "$OIDC_PROVIDER:sub": "system:serviceaccount:${EXTERNALDNS_NS}:external-dns",\
                       "$OIDC_PROVIDER:aud": "sts.amazonaws.com"
                   }
               }
           }\
       ]
   }
   EOF
   ```

IRSA_ROLE="nexusrepo-external-dns-irsa-role"
   aws iam create-role --role-name $IRSA_ROLE --assume-role-policy-document file://externaldns-trust.json
   aws iam attach-role-policy --role-name $IRSA_ROLE --policy-arn $POLICY_ARN

ROLE_ARN=$(aws iam get-role --role-name $IRSA_ROLE --query Role.Arn --output text)
   echo $ROLE_ARN
   ```

Take note of the `ROLE_ARN` output last above and specify it in your values.yaml for `serviceAccount.externaldns.role`.

#### External DNS YAML

After running the permissions above, run the [external-dns.yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-external-dns-rbac.yaml).

Then, in the [ingress.yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-ingress.yaml), specify the Docker subdomains you want to use for your Docker repositories.

#### [Step 9 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-9-validation--optional--161967_body)

**Note**
This section assumes the following:
- **Helm-Based Deployments** - You must have set `externaldns.enabled` to "true" in the values.yaml and installed the HA Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)).
- **Non-Helm Deployments** - You must have edited the [external dns yaml](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-external-dns-rbac.yaml) as appropriate for your environment and applied it to your cluster.

1. Ensure that you have an IAM role associated with a service account with the `AllowExternalDNSUpdates` permission and a trust relationship between the role and service account (you can find the trust relationship information under _IAM → Role → <Role Name> → Trust relationships_ in the AWS console). (Also see the [AWS documentation on how to use trust policies with IAM roles](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/).
2. Ensure that A records are created in the Route 53 hosted zone for the required Docker subdomains. (See the [AWS Route 53 documentation on working with hosted zones](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-working-with.html) and their [documentation on listing records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-listing.html).)
3. Use the AWS _Test record_ function in Route 53 to validate that the Route 53 records for the subdomains are returning "No Error." (See the [AWS Route 53 documentation for Checking DNS responses from Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-test.html).)
4. Ensure that the subdomain names you provided in the YAML filesor values.yaml (for Helm-based deployment) matches the subdomains that you want to configure in Sonatype Nexus Repository. This is the name you enter beside the _Allow Subdomain Routing_ checkbox as documented in our [Docker Subdomain Connector help documentation](https://help.sonatype.com/en/docker-subdomain-connector.html "Docker Subdomain Connector").

### Step 10 - Dynamic Provisioning Storage Class

Run the [storage class YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-storage-class.yaml). This storage class will dynamically provision EFS mounts for use by each of your Nexus Repository pods. You must make sure that you have an AWS EFS Provisioner installed in your EKS cluster.

Note that the Helm chart supports both EBS volumes and EFS mounts, but we strongly recommend using EFS. If you do use EBS volumes, you will need to use an EBS Provisioner.

#### [Step 10 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-10-validation--optional--161967_body)

In the AWS console, navigate to _EC2 → Volumes._ Ensure that the EFS mounts are provisioned. If they are not being provisioned, use the commands below to help determine possible causes:
- Describe your pod
   ```bash
   kubectl describe pod -n <your namespace> <pod name>
   ```
- Get the persistent volumes in your cluster
   ```bash
   kubectl get pv
   ```
- Describe a persistent volume in your cluster associated with your Sonatype Nexus Repository deployment
   ```bash
   kubectl describe pv <name of persistent volume>
   ```
- Get the persistent volume claims in your cluster
   ```bash
   kubectl get pvc -A
   ```
- Describe a persistent volume claim in your cluster associated with your Sonatype Nexus Repository deployment
   ```bash
   kubectl describe pvc -n <namespace> <name of persistent volume claim>
   ```

### Step 11 - AWS S3

Located in the same region as your EKS deployment, AWS S3 provides your object (blob) storage. AWS provides detailed documentation for S3 [on their website](https://docs.aws.amazon.com/s3/index.html).

#### [Step 11 Validation (Optional)](https://help.sonatype.com/en/option-3---high-availability-deployment-in-amazon-web-services--aws-.html\#step-11-validation--optional--161967_body)

1. In the AWS VPC console under _VPC → Endpoints_, ensure that you have a gateway endpoint for S3 in the VPC to avoid traffic having to go over the Internet. (See the [AWS documentation on gateway endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html).
2. In the AWS VPC console under _VPC → Route Tables_, ensure that the route table for the subnets has a route added to the S3 gateway endpoint. (See the [AWS documentation on configuring route tables](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html).

## Starting Your HA Deployment

At this point, you should have completed all of the steps for setting up your environment before actually starting Sonatype Nexus Repository. The steps below will actually start your deployment.

If you are using the HA/Resiliency Helm chart, you do still need to complete these steps. You will then need to follow the additional instructions in the Helm chart's Readme file.

### Step 1 - StatefulSet

Run the [StatefulSet YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-statefulset.yaml) to start your Nexus Repository Pods.

#### Step 1 Validation (Optional)

Expand Validation Steps

1. Use the following command to validate that there are four containers running within each pod for each replica:
   ```bash
   kubectl get po -n <namespace>
   ```
2. Execute the command below to ensure that Sonatype Nexus Repository is running without error:
   ```bash
   kubectl logs <pod-name-from-previous-step>   -n <namespace> -c nxrm-app
   ```

### Step 2 - Ingress YAML

Run the [Ingress YAML](https://github.com/sonatype/nxrm3-ha-repository/blob/main/sample-aws-ha-yamls/aws-ha-ingress.yaml) to expose the service externally; this is required to allow you to communicate with the pods.

#### Step 12 Validation (Optional)

Expand Validation Steps

After running the YAML/Helm charts, use the following command to validate that the Ingress for the ALB was created
   ```bash
   kubectl get ingress/ingress-nxrm-nexus -n <namespace>
   ```
   The output should display the ALB's DNS name.

## Sonatype Nexus Repository HA/Resiliency Helm Chart

**Note**
If you are using EKS version 1.23+, you must first install the AWS EBS CSI driver before running the HA/Resiliency Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)). We recommend using the EKS add-on option as described in [AWS’s installation instructions](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html).

Unless otherwise specified, all steps detailed above are still required if you are planning to use the HA Helm chart.

To use the HA/Resiliency Helm Chart ([GitHub](https://github.com/sonatype/nxrm3-ha-repository/tree/main/nxrm-ha), [ArtifactHub](https://artifacthub.io/packages/helm/sonatype/nxrm-ha)), after completing the steps above, use git to check out the Helm Chart repository. Follow the instructions in the associated Readme file.

**Note**
The Helm chart is set up to disable the `default` blob stores and repositories on all instances.