DevSecOps: Secrets in the Cloud

May 21, 2018
By Derek Weeks

6 minute read time

System operators managing security often (maybe always) worry about secrets used to access their networks and resources that will get out or be setup incorrectly, exposing their systems.

If this describes someone you know, you can ease their worries with system hardening. Akash Mahajan ( @makash) told his personal journey with system hardening in his talk, The Secrets in Our Clouds. Perhaps his experiences could help you implement or improve system hardening for your own system.

Akash is with Appsecco, which is a relatively small company with typical production and staging requirements:

1. They run multiple web applications with integrated authentication.
2. They run multiple websites with authentication and authorization.
3. In most cases, they terminate SSL/TLS at NGINX and reverse proxy to internal apps.
4. They generate TLS certificates using Let's Encrypt.
5. Whenever they add authentication as a middle-ware they rely on a SSO provider.
6. Their apps need access to certain secrets, such as passwords and authentication tokens.

Akash said he started his system hardening journey with, "what I know best: installing and configuring software." He chose Vault and Goldfish because of its well-written installation instructions, and most of the complexity is hidden. Additionally, it seemed easy enough to expand.

Once your software is installed, you have to answer where to store the secrets and what secrets do we need to worry about. Examples of secrets include:

• Certificates
• SSH keys
• API tokens
• Passwords (database, system, user, etc.)
• Anything else you need to store safely, such as audit logs that may contain sensitive details and TOTP generation keys.

After this was setup for Akash, he still had nagging doubts and gnawing fears in the back of his mind:

• Is it the best way to manage secrets?
• Should I use something cloud specific?
• How resilient is this?
• How will I automate to make it reproducible?
• What if I make a mistake in configuration and it leaks data out
• What if my backups fail?

What did Akash do? He switched hats from operations to security and didn't panic. He took a step back and thought of, "abstractions which are simple enough for me to understand." Some concepts that helped Akash include plaintext keys, secure storage, secure access, and the ability to choose which users see which secret.

Some attributes that helped Akash:

• If a digital secret is stolen, you still have your copy.
• The best way to store something important to you is encrypted, not in plaintext.
• If we encrypt something, there exists a key in plaintext.
• This plaintext key needs to be protected.
• This plaintext key should have secure storage, and we should prove that we are authorized to access it when we want it.
• Our plaintext key can be stored inside the Vault behind keys with multiple people.
• A cloud is someone else's computer. You can't store it in plaintext, but you can be rest assured AWS, Azure, etc. have done their security homework.

If you plan to store secrets in your cloud, Akash offered a few things to think about:

• Storage security - Can you expect better security than the cloud provider?
• Access security - Can your internal processes do better security than the cloud provider?
• Automation possibilities - If you do go ahead, what are the parts that will not be automated?
• Cooking required - If you plan to have a multi-cloud strategy, do you need to write your applications, automation scripts to have wrapper functions to ensure that you can plug-in/plug-out cloud provider specific code?
• Compliance - Do you have a specific compliance requirement which requires you to think differently about all this?

Akash wrapped up his talk with a recommendation to start simple, with two approaches for managing secrets: one for humans and another for systems.

For humans:

• Setup shared Keepass databases synced.
• Setup a Vault server behind Nginx and create entries for SSH keys and database passwords.
• Use a provider which can match company directory to make it easy to revoke access to an employee if required.

For systems:

• Use the same cloud provider where your servers are based.
• If possible, deploy a serverless app that exposes secrets both from the Cloud KMS and the Vault server.

If you are craving more on DevOps in Modern Infrastructure?
Binge watch any of the 20 DevOps in Modern Infrastructure sessions, free of charge, from All Day DevOps.

Written by Derek Weeks
Derek serves as vice president and DevOps advocate at Sonatype and is the co-founder of All Day DevOps, an online community of 65,000 IT professionals.