DevOps: Making the Boring Things Stay Boring

January 19, 2017
By Derek Weeks

4 minute read time

" I, For One, Welcome Our New Robot Overlords" is the title of Mykel Alvis' ( @mykelalvis) session at the 2016 All Day DevOps Conference. He wasn't trying to curry favor with the new robot rulers, a la Kent Brockman, but instead was evangelizing on the importance of precision in DevOps.

Mykel is the DevOps Coach at Cotiviti Labs, the development arm of Cotiviti ( http://www.cotiviti.com/). Every day they handle healthcare data, which is one of the most regulated and protected data around. As Mykel said, "We don't do cat pictures on the Internet." A mistake, breach, security flaw, etc. could mean the exposure of privacy and financial data for millions of people. But, they are a large organization, with exposure that needs to be managed. They have 200-plus repositories, 3-4 deployments per day, 15 developers, and 0 operators (sort of). Without the right discipline and rigor, costly mistakes could happen.

Hence, Mykel's need for the robot overlords. As he noted during the conference session, robots have "lower utility, but greater predictability." They do exactly what they are told, and they do it the same way, every time. By contrast, humans are superior to responding to unpredictable events. Mykel's goal is to leverage the strengths of each. Cotiviti Labs has systems, policies, procedures, and a culture in place that supports the automation of tasks as much as possible to ensure mistakes are kept to a minimum and humans can focus on the unpredictable. They are striving to make the boring things stay boring.

They call it "The Labs Way" and here is how they do it:

Treat everything as code. Everything. Infrastructure, data, everything. As an example, they perform pull requests to get status updates for executive reports.
Test the code.
Perform formal releases. They build their code in very specific ways, with extremely rigid gatekeeping processes.
Releases produce immutable artifacts. Once something has been released, they will never change it. They could, but they don't.
Keep everything. Not in a huge, unorganized file system. In an organized repository.
Failing tests mean a failed build.
Design your systems to be automated. They don't do things that can't be done automatically.
Deal with those systems only through automation. This is probably the most challenging thing we do, since it is extremely hard to convince developers to write their code to do this.
Operations IS developing. Operations is not a bucket for development. Operations is a development task. You treat it like it is real code.
Because everything is code. Everything.
All defects are defects of code.

To achieve this, they:

Standardize.
Evaluate.
Test.
Discard the useless.
Own it.

None of this comes easy. It challenges norms. It pushes boundaries. It mandates discipline. (Pull quote: "Discipline is the bridge between goals and accomplishment." - Jim Rohn) Your team must understand that you have a way to do things. There is a reason, and it will reduce risks and ensure everything works. It will keep the boring, boring.

The presentations from the All Day DevOps Conference are available online, free-of-charge here.

Written by Derek Weeks

Derek serves as vice president and DevOps advocate at Sonatype and is the co-founder of All Day DevOps, an online community of 65,000 IT professionals.