5 Quick Wins for Securing Continuous Delivery

October 04, 2018
By Derek Weeks
6 minute read time

" DevOps is Agile on steroids - because Agile isn't Agile enough."

So says Jim Bird, the CTO for BiDS Trading, a trading platform for institutional investors. Jim continued, "DevOps teams can move really fast... maybe too fast? This is a significant challenge for operations and security. How do you identify and contain risks when decisions are being made quickly and often by self-managing delivery teams? CABs, annual pen tests, and periodic vulnerability assessment are quickly made irrelevant. How can you prove compliance when developers push their own changes to production?"

Jim was presenting at the 2018 Nexus User Conference on continuous delivery. Pulling on his 20+ years of experience in development, operations, and security in highly regulated environments, Jim laid out how and why continuous delivery reduces risk, and how you can get some easy wins toward making it more secure.

As Jim points out, it reduces risk because:

• Breaking work down and delivering it incrementally reduces delivery costs and risks.
• Continuous delivery takes care of the critical "last mile" of production.
• Change becomes routine.
• Change becomes cheap, because teams can afford to make smaller bets, iterate, and learn.
• Change becomes fast. Teams can quickly push fixes to respond to failures or security threats.

To ensure continuous delivery is secure without getting in the way of the delivery team, Jim outlines a few key points:

• Understand the team's workflow and tool-chain, work the way they work.
• Address real risks and threats, problems that everyone agrees are important and relevant.
• Take an incremental, iterative approach.
• Provide useful, quick, and continuous feedback to the team and management.
• Minimize friction and delays.
• Look for controls and checks that are easy and cheap to implement.

With the case made, Jim provides five quick wins for continuous delivery.

Automated Safety Checks in Your Pipelines

Why? Because you often make changes, it is easier to make mistakes. You need simple, fast checks that find them before the bad guys. This makes auditors happy, because they can see that policies are being enforced.

How? Add tests/assets for common mistakes to the end of the automated build/deploy pipeline in test and production. This includes: default/missing credentials; SSL and HTTPS settings; known critical vulnerabilities; file and user permission, and open ports.

Tools and frameworks available include:

• Netflix Security Monkey, Conformity Monkey
• Gauntit
• OWASP ZAP Baseline Scan
• OSQuery
• InSpec
• Cloud services like AWS Inspector

Lightweight Risk Assessments

Why? Teams are making decisions quickly, so you want them to do this safely and look for potential new risks upfront before coding starts. You get security/compliance stories into the backlog so they can be planned for, and so you can find the right tools, libraries, and frameworks. It also makes the risks visible to the teams and management.

How? Check out Mozilla's Rapid Risk Assessment. It's a good example.

Security Unit Tests

Why? These prove vulnerabilities are being fixed correctly. They teach developers how to "step off the happy path" in their testing, and it will catch/prevent regressions. Normal tests prove it works, not that it might not work, and developers need to write happy tests, sad tests, and bad tests.

How? As you find vulnerabilities, write a test first, so you can prove it exists. Then, have developers fix it and then run the test again. They can use tools such as JUnit, NUnit, and Jasmine.

Secure the Software Supply Chain

Why? Third-party code injects risk and security vulnerabilities and technical debt: known security vulnerabilities; abandoned and out-of-date code; and license risks. It is relatively easy, there is some lifting at the front, but once you do it, you have a nice control set going forward.

How? Get a tool to automatically catalog dependencies and check against vulnerability databases. Integrate it into the continuous delivery pipeline, and fail builds on serious problems.

Tools include:

• OWASP Dependency Check
• Retire JS
• Anchore Engine
• Sonatype Lifecycle
• Sonatype Repository Firewall
• Licensed
• GitHub Security Alerts

Self-Service Code Scanning

Why? Automated static analysis scanners can catch common coding mistakes and bad practices. This is the only way to cover large code bases, and it backstops code reviews, because reviewers won't waste time on superficial and subtle coding problems. It fits naturally into workflows, and results can be directly fed back to developers.

How? Focus on high-risks and high-fidelity rules. Accuracy and speed are more important than completeness in continuous integration/continuous delivery.

Tools include:

• Java - SpotBugs + FindSecurityBugs plugin + FBContrib plugin
• .NET - Puma Scan, Roslyn Security Guard, and Microsoft's DevSkim
• SonarLint IDE plugin for various languages
• Ruby on Rails - Brakeman
• Docker - Hadolint
• Git Pre-commit checks to catch secrets, etc. on check in: Talisman, Yelp's pre-commit, git-secrets
• Git tools to scan reports for secrets: TruffleHog, git-all-secrets
• A list of static analysis tools

While Jim's quick wins reduce risks and get the ball rolling, he emphasizes that "deep, real change to culture and how people work takes a long, long time, patience, and reinforcement."

Written by Derek Weeks
Derek serves as vice president and DevOps advocate at Sonatype and is the co-founder of All Day DevOps, an online community of 65,000 IT professionals.