Using S3 as a File Store for Self-Hosted IQ Server

Self-hosted Sonatype IQ Server can use Amazon S3 or an S3-compatible service as external storage for scan data, reports, and SBOMs. This offloads large artifacts from local disk to durable, scalable object storage.

IQ supports Amazon S3 and S3-compatible object storage implementations that are fully compatible with the AWS S3 API and the AWS SDK for Java 2.x. More detailed requirements are explained below in Prerequisites.

Because S3-compatible implementations can vary in behavior and performance characteristics, we recommend validating compatibility and performance with your storage vendor for your expected workload.

Note

S3 file store support for self-hosted IQ Server is available starting in IQ Server release 196.

By default, IQ Server stores all data on the local filesystem under the sonatype-work/clm-server directory. With S3 storage enabled, the following artifact types are stored in S3 instead of on disk:

All other data (e.g., logs, the database (if using H2), temporary files, search indexes, and application icons) remains on the local filesystem.

Prerequisites

To use S3 as a file store in self-hosted IQ environments, you must meet the following prerequisites:

• An S3 or S3-compatible bucket in the desired AWS region

• IQ supports Amazon S3 and S3-compatible object storage implementations that are fully compatible with the AWS S3 API and the AWS SDK for Java 2.x.

• When using an S3-compatible provider, configure the service endpoint using the endpoint override setting.

• IQ currently requires virtual-hosted-style bucket addressing (for example, https://my-bucket.s3.us-east-2.amazonaws.com/object-key). Path-style addressing (for example, https://s3.us-east-2.amazonaws.com/my-bucket/object-key) is not currently supported.

• AWS credentials available to the IQ Server process

• For existing installations, you must stop IQ Server before changing the storage configuration

New vs. Existing IQ Server Installations

For new IQ Server installations, simply add the storage block to your config.yml as explained in the Configuration section below before first startup. IQ Server will write all scan, report, and SBOM data directly to S3 from the start.

If you have an existing IQ Server installation using local storage, see Migrating Existing IQ Server Data to an S3 File Store below.

Performance Considerations

Before deciding to use S3 for your IQ Server file storage, consider the following:

• S3 storage introduces network latency for read and write operations. Ensure adequate network bandwidth between IQ Server and the S3 endpoint.

• Uploads over 100 MB are automatically handled using S3 multi-part uploads with a minimum part size of 5 MB.

Configuration

To configure Sonatype IQ Server to use S3 as a file store, begin by adding the following storage block to your config.yml:

storage:
  type: S3
  s3Config:
    bucketName: "my-iq-bucket"
    region: "us-east-2"

The following properties can be configured within this block:

Below is a complete example of a storage block in the config.yml for configuring IQ to use AWS S3 as a file store:

storage:
  type: S3
  s3Config:
    bucketName: "my-iq-bucket"
    region: "us-east-2"
    objectKeyPrefix: "production"
    serverSideEncryption: "AES256"

With this configuration, objects will be stored under keys like production/scan/{appId}/…, production/report/{appId}/{scanId}/..., etc.

To use an S3-compatible service, set the endpoint property to the service URL as in the following example:

storage:
  type: S3
  s3Config:
    bucketName: "my-iq-bucket"
    region: "us-east-2"
    endpoint: "https://example.example.com:9000"

Authentication

Sonatype IQ Server uses the AWS Default Credentials Provider Chain to authenticate with S3. Credentials are resolved in the following order:

1. Java system properties – aws.accessKeyId and aws.secretAccessKey

2. Environment variables – AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

3. AWS credentials file – Shared credentials file (i.e., ~/.aws/credentials)

4. IAM role – Instance profile credentials (EC2) or container credentials (ECS/Fargate)

You cannot configure AWS credentials in the config.yml. Instead, supply credentials through one of the methods above.

Example: IAM Role (Recommended for AWS Deployments)

If IQ Server runs on an EC2 instance or in ECS/Fargate, attach an IAM role with the necessary S3 permissions. No credential configuration is needed; the AWS SDK discovers the role automatically.

Example: Environment Variables

The example below shows how to configure credentials via environment variable:

export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Required IAM Permissions

The IAM principal used by IQ Server needs the following permissions on the configured bucket:

{
  "Version": "2012-10-17",
  "Statement": [\
    {\
      "Effect": "Allow",\
      "Action": [\
        "s3:GetObject",\
        "s3:PutObject",\
        "s3:DeleteObject",\
        "s3:ListBucket",\
        "s3:HeadObject"\
      ],\
      "Resource": [\
        "arn:aws:s3:::my-iq-bucket",\
        "arn:aws:s3:::my-iq-bucket/*"\
      ]\
    }\
  ]
}

If using KMS server-side encryption (aws:kms), the principal also needs kms:GenerateDataKey and kms:Decrypt permissions on the KMS key.

Kubernetes Deployments

For Kubernetes deployments, the storage configuration is set in config.yml as described in the Configuration section above.

AWS credentials are typically provided through one of the following means:

• A Kubernetes service account with IAM role annotation (recommended for EKS)

• Environment variables injected from Kubernetes Secrets

• Mounted AWS credentials files

Migrating Existing IQ Server Data From Local Storage to an S3 File Store

If you have an existing IQ Server installation using local disk storage, you can migrate scan, report, and SBOM data to S3. Note that migration requires a maintenance window while IQ Server is stopped. Note that expected transfer time depends on the amount of data you have to transfer; it may take considerable time to transfer large amounts of data.

To perform the migration, take the following steps:

1. Stop IQ Server

Shut down IQ Server to ensure no data is being written during migration.

2. Sync data to S3

Use the AWS CLI to copy the relevant directories from sonatype-work/clm-server to your S3 bucket as in the following example:

   aws s3 sync ./sonatype-work/clm-server/ s3://my-iq-bucket/ \
        --exclude "*" \
        --include "scan/*" \
        --include "report/*" \
        --include "sboms/*" \
        --sse AES256

If you configured an objectKeyPrefix, include it in the destination path as in the following example:

   aws s3 sync ./sonatype-work/clm-server/ s3://my-iq-bucket/production/ \
        --exclude "*" \
        --include "scan/*" \
        --include "report/*" \
        --include "sboms/*" \
        --sse AES256

3. Update the config.yml and ensure authentication is in place

Add the storage block to config.yml with type: S3 as described in the Configuration section above. Also, ensure IQ Server still has credentials available as described in the Authentication section above.

4. Start IQ Server

Start IQ Server; it will now read and write scan, report, and SBOM data from S3.

5. Verify and Reclaim Disk Space

Run a scan and policy evaluation to confirm reports are stored in S3. You can verify by listing bucket contents as in the following example:

   aws s3 ls s3://my-iq-bucket/scan/ --recursive | head
   aws s3 ls s3://my-iq-bucket/report/ --recursive | head

After verifying that data is being stored in S3, you can safely remove the local scan/, report/, and sboms/ directories under sonatype-work/clm-server to reclaim disk space.

What Remains on Local Disk?

Even with S3 storage enabled, the following data stays on the local filesystem under the sonatype-work/clm-server directory (or the path configured by sonatypeWork):

The PostgreSQL database is managed separately and is not affected by this configuration.

Scans, reports, and SBOMs typically account for the majority of disk usage. Moving them to S3 significantly reduces local storage requirements. The remaining items listed above are generally small and low-volume.