DevOps: Building Better Pipelines

March 06, 2018
By Derek Weeks
3 minute read time

"The deployment pipeline takes value from development and delivers it to clients."

This is one of the better summations of a development pipeline that I have heard, and it came from Dan Barker ( @barkerd427) during his recent All Day DevOps conference talk, " Becoming a Plumber: Building Deployment Pipelines." Dan's talk focuses on the value of pipelines, how to build them well, and how to build both development and infrastructure pipelines.

Traditional Pipelines

The traditional pipeline:development to testing to production servers is a good foundation, and, like the plumbing in your home, is helpful to generally get us where we want to go. However, as anyone who has raised boys knows, a toilet doesn't guarantee 100% compliance. The user isn't perfect.

It is similar to basic development pipelines. Things don't go as planned. As Dan mentioned in his talk, he strides to keep his code all in dev, and then push to test, and then push to production. However, that doesn't always happen. And there are other inconsistencies that inadvertently crop up (such as tab vs. space), despite everyone's best intentions.

Dan is a chief architect and works in highly regulated industries: financial services, health care, and insurance. So, precision, compliance, audit trails, etc. are all the more important. He architects pipelines to ensure human error is minimized.

What's in a Pipeline?

But why all the fuss over pipelines? Well, first Dan touched on the values pipelines provide:

Abstract audit and compliance.
Trivialities eliminated (spaces vs. tabs).
Security checks occur early/often.
Tests all the things.
Nimble security.
Common artifact repositories.
Standardized approval system.
Apps become secure by default.

But how do you architect a pipeline to deliver these values?

In the illustration below, you can see Dan's pipeline architecture, where he offers key points that all of us should consider applying:

Production data is separated from development data. You can only get there through the pipeline.
You interact with the Platform, and it configures the levels below it.
You can duplicate containers as much as you want to scale horizontally, and can replicate that in production.
The configuration must come from the environment.

As he dug in, he started with covering the advantages of two types of pipelines, recognizing you need to select this first:

Scripted:

Very Groovy.
More powerful.
Provides the greatest flexibility.

Declarative:

Only a little Groovy.
Simpler to maintain.
Easier to read and understand.

Dan then dug into the technical nitty-gritty, such as shared libraries, setups, software to manage pipelines, fabric8, Kubernetes, Yahoo! Screwdriver, etc., and then made the observation that he thinks deployment pipelines have fallen behind. In his real-life example, he and his team decided to take a more holistic approach to their pipeline (think: the first way of DevOps) so they could build and deploy apps, and also build and deploy the infrastructure.

Sharing Knowledge

They built a system that is especially applicable to those in highly regulated industries, and they hope to open source it soon. Keep an eye on @barkerd427 for more info. In the meantime, he describes in detail the process it follows, software and libraries it uses, etc.

Yearning for more? You can watch any of the 2017 All Day DevOps sessions free-of-charge here.

Written by Derek Weeks

Derek serves as vice president and DevOps advocate at Sonatype and is the co-founder of All Day DevOps, an online community of 65,000 IT professionals.