How to Design Teams to Bridge the Security Gap

April 17, 2018
By Manuel Pais

11 minute read time

Even if you work for a large organization, chances are you could memorize the names of all the security folks working in IT. That's because the ratio of developers to security is 100:1, according to a recent survey (the same study indicates a 10:1 ratio of devs to ops). Previous studies have reported ratios from 100:3 to 100:6, so there's some progress but not fast enough.

That doesn't bode well for these organizations' ability to tackle upcoming data privacy protection regulations like the European Union's GDPR and stop the data spilling regularly on the news over the past few years. There aren't enough skilled security folks out there today to fill in the gap. Successfully adopting secure development and operations practices requires deep technical understanding, but also critically possessing the communication, persuasion and balancing skills to achieve buy-in from development teams and (more often than not) senior management.

We're still in a vicious cycle where security specialization is not attractive enough, because historically, companies have not given security enough priority. Those same companies are now facing a growing need to up their security game, but they lack enough people with the right skills.

So what options do we have to bridge this security gap?

Let's first clarify that automating basic security checks (such as third-party vulnerabilities) and coding guidelines and integrating security in the delivery pipeline should be a no-brainer for any product development team today. The toolset in DevSecOps keeps expanding, and both community and enterprise-grade requirements are well served.

So where's the gap? It's in the "hacker mindset," a constant search for new potential attack vectors in our applications (the "unknown unknowns"). And it's also in the hard task of meaningful risk analysis and threat modeling and its translation into actions and priorities. These require deep security thinking and background, but it's not something product development teams can be asked to take on as added cognitive load on top of their existing responsibilities.

The work Matthew Skelton and I have done researching, cataloging and explaining how different DevOps Topologies can play an important role in progressing or blocking DevOps adoption fits nicely with the problem at hand.

Two distinct (and possibly complementary) team topologies come to mind:

• Fully shared security responsibilities.
• Security as an enabling team.

Fully Shared Security Responsibilities

Fully Shared Security Responsibilities means each product development team integrates at least one security-focused T-shape team member. This approach is adequate when the organization strives for cross-functional autonomous product development teams. The security person needs to translate complex security threats into actionable security stories and acceptance criteria that the team can understand and implement from a technical standpoint.

They must also communicate risks and potential costs for the organization (compliance, revenue and reputation) in a way that business owners can understand and prioritize. On the other hand, they should be ready to perform non-security related tasks when needed. Over time, ownership of security analysis and implementation should spread in the team, while the T-shaped security person might maintain expertise in terms of staying on top of security best practices, new methods and tools, facilitating security-related workshops and leading product security strategies.

The goal is not for the security person to be allocated all the security work, otherwise we are just moving a silo at a macro level (isolated security team) to a micro-level (isolated team member).

Each product team - depicted as a yellow circle - with 7 to 9 team members, where at least one is a T-shaped security person - depicted inside a green circle - but other people also participate in security work.

Of course, there is still an important place for a centralized view of security across multiple products and business areas in the organization. Sharing experiences, approaches and results is critical. But this can take the form of a community of practice or a guild (in Spotify parlance) of motivated individuals gathering regularly, rather than a dedicated team.

Pros

• Team security ownership will rise, leading to faster (non-trivial) security incident resolution, and perhaps more importantly, learning from them to avoid repeating the pain in the future.

• As teams have a natural size limitation (8-9 people), the ratio of IT staff to security (T-shaped) staff should move much closer to 10:1 over time with this topology. The diversity in technical background alone will bring benefits in terms of approach to problems and priorities.

• The move to this topology will send an undeniable signal to everyone in the organization that security is now a first-class citizen in our products.

Cons

• The cost of finding and recruiting (which might require generous packages) and ramping up those extra nine security persons (in an org with 100 developers) will be a serious issue (as per the intro to this post). Expect this transition to take considerable time, during which you need to have a mix of models (some autonomous product teams and some still depending on a centralized team), and think about a strategy for selecting which teams go first. For example, start by assigning the more experienced security folks to the teams working on the riskier products in your organization.

• High-performing autonomous teams are based on stability and knowing each other's strong and weak points. Any change to the team composition, even a single person, will inevitably cause some disruption. The team might feel it is delivering less and becoming less productive while taking in the new security angle. It is crucial that everyone understands that this is normal and accepted.

• Lack of a centralized contact point for security concerns. Senior managers, in particular, might feel there is "no one at the wheel of security" in a decentralized structure. Making sure there are communication channels and people can direct requests for security information to the right teams (or to a community of practice) can help here (although that could work, if you have the means).

Heuristics

• Does your organization already have cross-functional teams, integrating business owners and owning responsibility for QA, monitoring and running the applications they develop?

• Do products (or services) display very different risk profiles?

• Do products (or services) run on heterogeneous tech stacks and infrastructure?

If the answer to one more of the above questions is affirmative, then this topology might be suitable to address the security gap.

Security as an Enabling Team

Security as an Enabling Team means a dedicated security team provides support and guidance (for example, producing secure development guidelines) to product development teams.

Crucially, this centralized team does not perform the actual security analysis and implementation work, but promotes and guides it where necessary.

It's equally critical that this team gets involved from the start of the project or release, in order to help the product team consider the security implications, approaches and practices at each stage of the life cycle.

A transversal security team - depicted vertically in gray - supports three product teams - depicted horizontally in orange - resulting in shared responsibility for security - depicted as a light green circle.

Many organizations have taken this approach, including Spotify and Sportradar. It requires a much less drastic structural change from the traditional "security team silo," as we're essentially shifting this team's responsibilities (guidance and support rather than implementation). But that might make it harder for the rest of the organization to fully realize that product teams are expected to take greater ownership of their systems.

Pablo Jensen, CTO at Sportradar, recently talked about their dedicated information security team's role in promoting security guidelines and policies in close collaboration with product teams, listening to their feedback and iterating over time. One of their project life cycle gates is a "fitness to start development," which requires the security team to approve that enough thought has been given to security implications and work planned accordingly. This team reports directly to the CEO and has authority to stop product launches if necessary.

Pros

• Shorter timeframe to move to an enablement mode from a traditional isolated security team doing the bulk of the security work.

• Will not require many new hires, thus avoiding all the associated effort and potential team disruption. The focus should be on ensuring that the enabling team has all the soft skills required, on top of technical skills.

Cons

• This team needs to master effective communication, which is a good thing. This is a skill that needs to be honed over time. Therefore, initial investing in external or internal help define communication strategies and curate content.

• Moving to this model sends a weak signal about the change in security focus. Signal needs to be amplified, and might require repeating for a long time until it sinks in as part of the org culture.

• Introducing new responsibilities, practices and tools can be challenging for product development teams already at the peak of their capacity. The centralized enablement team might get overwhelmed with requests for help and be put under pressure to help with product security implementation, which would defeat the entire purpose of the model.

• Over time the security focus inside product teams might fade away without a security person participating in the team's release/sprint planning and daily standups. Some kind of governance will need to be put in place to avoid this effect.

Heuristics

• Is there a small number of product development teams (allowing for closer collaboration with the security team)?

• Do the product team members display interest in security topics and appetite to learn (for example attending technical conferences or meetups)?

• Does the resolution of security related incidents in production naturally involve both security and development people (as opposed to a blame game where each side shies away from responsibility)?

If the answer to one more of the above questions is affirmative, then this topology might prove suitable to address the security gap.

Conclusion

DevSecOps has raised the profile of security in IT, and the regular stream of serious data breaches has exposed the security gap in most organizations. In this post, I presented a few possible approaches to bridge that gap (fully shared security responsibility vs security as an enabling team), along with their pros and cons, and heuristics based on context.

Remember, team design should not be static. Organizations, people and processes evolve naturally over time, and what today's best fit might be tomorrow's obstacle to faster and safer software. It's possible that an organization would first move from a traditional security silo approach to a "security as enabling team" model at first, and then transition to a "fully shared security responsibilities" structure, as security awareness and expertise spreads across the product teams.

What other team topologies and security strategies have you seen in your organizations or clients? Please email me at: me at manuelpais dot net.