Data Management and Storage Best Practices
Data Management and Storage Best Practices
The IQ Server is fairly lightweight, but managing your IQ Server's disk usage is important because:
- Storage can be expensive, especially if you're in the cloud.
- Your storage needs will grow as your usage of IQ Server matures.
- A lean IQ Server makes backing up and restoring faster and easier.
Follow these best practices to manage your IQ Server's data efficiently.
Deploy IQ Server into an environment with an appropriate amount of disk space
Sonatype recommends that you have between 500 GB and 1 TB of storage available for the IQ Server.
Your disk usage is largely a product of (the number of apps x the number of scan events). Therefore, scanning lots of apps frequently will fill up disk space fast.
Customers who rely heavily on SCM integrations may benefit from more storage. Use sparse checkout file types with your SCM integrations to address this.
To maximize your storage space, get the best performance, and for overall ease of management, use the Postgres database option.
Follow your organization's data retention guidelines
Your organization may have guidelines for storing assets long-term. Always heed and follow those guidelines.
If you don't have data retention guidelines, consider putting some in place for IQ Server assets specifically.
Reports for internal dev builds typically become irrelevant after a few months.
Reports for a release/production version of your app should be kept for longer.
Have a way for stakeholders to protect specific data
- Some stakeholders may require specific data. For example, your Legal team may want to hold on to some key reports forever. Make sure you know what your stakeholders need, and have a way for them to request that some files never get deleted.
Work with your database administrator
- Large-scale cleanup efforts should include your database administrator. Mismatches between your sonatype-work directory and your database can cause hiccups in the browser UI.
Have a plan for long-term storage
If you need to save a report for longer than 5 years, move it out of your work directory.
Reports can be transformed into SBOMs for more efficient storage.
Reports may contain sensitive information, including things like application names, component names, file extensions, and vulnerability data.
Configure Data Retention
Data Retention handles much of the regular cleanup associated with managing an IQ Server for you. It's on by default—leave it on!
Data Retention can trigger based on age or based on the total number of reports in a given stage. Consider what stages you generally scan at, and configure accordingly.
- For example, if you rely heavily on SCM integrations, you'll likely have many scans at the Source stage. In that scenario, make sure the number of maximum reports at the Source stage is high enough that new reports aren't deleting old reports that you still need.
Purged files are placed into a separate trash folder. By default, this is sonatype-work/clm-server/trash.
Files in the trash folder are not deleted automatically – they must be deleted manually.
Whenever convenient, check the trash folder and delete the files therein.
The contents of your trash directory may contain sensitive data. Compress reports still contain things like application names, component names, file extensions, and vulnerability data.
Use sparse checkout for your SCM integrations
- Sparse checkouts of your SCM repositories only clone the files that IQ Server requires in order to perform a full scan. This can dramatically decrease the amount of disk space your IQ Server requires.
Know what to save and what to delete
Some files can be more safely deleted than others.
Success Metrics is a measure of your IQ Server usage over time, so they should not be deleted or moved from the sonatype-work directory. Try to save them at least once a year.
Reports may be okay to delete. Follow your organization's data retention rules, but remember that reports are a point-in-time record of your app's contents. Your most recent report is always the most important report.
New log files eventually replace old ones. Check the Config YAML file to see how many archived log files are kept, and adjust up or down as needed. The default is 50.
SCM files can be deleted safely; if the IQ Server needs them again, it will download them again. However, this could cause issues with your SCM repository's rate limit, if it has one, so exercise caution.
Your Advanced Search index (sonatype-work\clm-server\search) is required for Advanced Search and the Advanced Search API. If it's deleted, you can reindex, but this will take some time.
New scan files replace old ones, and each app only has one scan file at a time. This means that efforts to clean your scan folder aren't usually worth it.