2015 State of the Software

Supply Chain Report:

HIDDEN SPEED BUMPS ON THE ROAD TO “CONTINUOUS”

Foreword by Gene Kim, Gareth Rushgrove, John Willis, Jez Humble, and Nigel Simpson

TABLE OF CONTENTS

Foreword .....3 Introduction .....5 Why All Modern Software Development Relies on a Software Supply Chain .....6 SUPPLIERS: Open Source Projects .....7 Public Repositories (The Warehouses) .....8 Choosing the Best Suppliers (Sourcing) .....9 PARTS: Open Source Components .....12 Repository Management (Local Warehouses) .....15 MANUFACTURERS: Assembled Software Development .....19 Technical Debt: Assembly Line Inefficiencies .....21 FINISHED GOODS: Software Applications .....22 The Volume of Elective Re-work and Risk .....22 Software Bill of Materials .....23 Quality Controls: OWASP, PCI, FS-ISAC, U.S. Congress .....23 Lessons Learned from Traditional Manufacturing Supply Chains .....25 Automation: How To Improve Software Supply Chains .....26 Appendix .....28 Figure 1: The Volume and Size of the Global Software Supply Chain Figure 2: Target Benchmarks for Software Supply Chain Practices - Quality Control Figure 3: Target Benchmarks for Software Supply Chain Practices - Efficient Distribution Figure 4: Analysis of Components Used within Applications Figure 5: Multiple Versions of Parts Often Downloaded by the Largest Development Teams Figure 6: Volume of Defective Parts Used Figure 7: Comparison of Impact of Supply Chain Complexity on Prius versus Volt Figure 8: Efficient Sourcing Practices By Manufacturers

FOREWORD

“Just as in manufacturing, the effective management of our supply chains will create winners and losers.”

Gene Kim, Co-author of “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” and upcoming “DevOps Cookbook”

“Anyone who believes, as I do, that we can learn valuable lessons from manufacturing and supply chains on how to better manage technology work will love this report. To describe how we assemble and integrate open source software into the services we create, Sonatype uses the metaphor of the ‘software supply chain. ’ This metaphor enables some startling revelations on how we should select the components we use and the downstream effects of the decisions we make.

“As the custodians of the Central Repository, the largest open source repository in the world, Sonatype has insights into how the largest software supply chains in the world are managed— they’ve analyzed how over 106,000 organizations use over 1 million open source software components, spanning billions of component downloads.

“Just as in manufacturing, the effective management of our supply chains will create winners and losers. This will impact the quality of the services we deliver to our customers, as well as our ability to secure and maintain those services. ”

“This report provides vital insight into the state of our software supply chain.”

Jez Humble, Vice President at Chef, co-author of “Continuous Delivery and Lean Enterprise”

“The use of open source software components has become pervasive in the enterprise -- and rightly so. This reuse reduces complexity and time-to-market while increasing the security and reliability of software when done right.

“However, the research shows us many organizations are not doing it right. The result is a profusion of complexity in production systems that leads to poor quality, unreliable services as well as providing a rich target for bad actors.

“This report provides vital insight into the state of our software supply chain and practical advice on how organizations can innovate at scale while optimizing quality and security. ”

“This report draws parallels with traditional manufacturing supply chains.”

Nigel Simpson, Director of Enterprise Architecture, Fortune 100 Media & Entertainment Company

“This report draws parallels with traditional manufacturing supply chains, giving us a new way to look at how we build software. With this fresh perspective we can see the importance of effectively managing the software supply chain, especially when software engineering is becoming increasingly commoditized through the use of reusable, off-the-shelf, open source components.

“With visibility into the supply chain, we’ve identified many ways to streamline development and reduce risk. Just as in the automotive industry, where use of flawed components such as airbags has downstream impact on vehicles spanning multiple brands, we discovered that standardization of flawed open source components can impact hundreds of applications. Understanding the composition of our mission-critical applications has never been more important. ”

John Willis, DevOps Days core Organizer and co-author of the upcoming “DevOps Cookbook”

”Deming was one of the original prophets of Supply Chain hygiene. It’s great to see this subject being discussed related to the software industry and more importantly dealing with Open Source.

“Moreover, in a world that is vastly moving to containers and immutable infrastructure, the subject of Software Supply Chains is going to become of increased importance. “

“In a world that is vastly moving to containers and immutable infrastructure the subject of Software Supply Chains is going to become of increased importance.”

“This report is

required reading for anyone interested in large-scale systems engineering.”

Gareth Rushgrove,

Senior Software Engineer, Puppet Labs / Curator of “DevOps Weekly”

“It’s easier than ever to build complex systems quickly using open source components downloaded from the Internet. But where does that software (and its dependencies) come from? How do you keep it up to date? Is it introducing a critical security flaw to your application? The move towards microservices and polyglot programming environments makes these issues even more pressing, and the number of third-party components has grown too large to manage in a non-systematic way.

“This report introduces approaches that can be used to mitigate the risks of poor software supply chain management. Better still, the discussion focuses on techniques that have been successfully applied in other industries, along with hard numbers that show the size of the problem. This report is required reading for anyone interested in large-scale systems engineering. ”

INTRODUCTION

Organizations continually strive to produce quality software even faster and more efficiently. Over the past decade, software development practices have witnessed significant changes that have greatly improved velocity, agility, and innovation. Key among these is the reliance on open source or third party component “building blocks” freely available from a wide variety of sources. As a result, 80-90% of a typical application is made up of components used to quickly add valuable features without custom coding¹. The use of open source as well as other build components has led to the creation of a software supply chain.

While open source components and software supply chains are delivering huge benefits in terms of speed, the free-for-all nature of component availability and consumption practices also have led to significant waste, lower quality, and increased drag often not recognized by CIOs, enterprise architects, DevOps leaders, and software development teams. automation should be the new aim for software To begin improving the software supply chains that feed our software development practices, organizations first need to recognize that they exist and see the inefficiencies that are often hidden from view.

As the steward of the Central Repository, Sonatype watches over the world’s largest public open source repository every day. Our role as steward enables us to provide a detailed perspective on the suppliers, consumption volumes, distribution mechanisms, and quality attributes of open source components that flow across today’s software supply chains.

Our study of Central’s use by more than 106,000 software development organizations in 2014 astounded us—so much that we wanted to share our analysis in this first-ever report on the State of the Software Supply Chain. We discovered that current practices are silently sabotaging a

software development organization’s efforts to accelerate development, improve efficiency and maintain quality. In short, our dependence on open source software components is growing faster than our ability to effectively source, manage, and secure it.

As you read this report, you’ll discover eye-opening statistics on the usage of open source components. While this report offers sobering news about the state of the software supply chain, it also adds perspective on how a number of organizations have taken steps to improve their software supply chains. We need not reinvent the wheel. We need to recognize that traditional supply chain principles also apply to software development and can have the same transformative effect.

Given this evidence and the drive towards continuous and DevOps practices, software supply chain development organizations that want to make the world’s best software, responsibly and profitably. This aim is simple and achievable. There are lessons that can be learned from traditional manufacturing supply chains to help organizations overcome the challenges identified through this research:

1. Use fewer and better suppliers
   

2. Use only the highest quality parts
   

3. Track what is used and where
   

We have seen enterprises apply these principles and boost developer productivity from 15 - 40% by:

• Reducing unplanned, unscheduled rework

• Reducing the mean-time-to-remediate issues

• Reducing the number of known defective parts

• Reducing mountains of technical debt

• Reducing complexity that leads to maintainability issues

WHY ALL MODERN SOFTWARE DEVELOPMENT RELIES ON A SOFTWARE SUPPLY CHAIN A View Across 106,000 Organizations

In traditional manufacturing, we have suppliers, parts, warehouses, manufacturers, assembly lines and finished goods. In software development we have uncanny similarities. Yet software supply chains lack the rigor and processes that have fueled high-velocity production, operational efficien- cies, and competitive differentiation witnessed in other supply chains.

If your company develops software, you’re likely consuming thousands of open source and proprietary components. Although your aim is to produce the highest-quality software in the most efficient way, a closer look at the statistics shows a potentially different story. By understanding the software supply chain operating—largely hidden—at the core of your operations, you can

more easily make small changes that yield dramatic gains.

As the steward of the Central Repository, the largest public open source repository, Sonatype has amassed a vast amount of data on the size, scale, velocity, and patterns of open source consumption across common software supply chains.

You’ve got ... SUPPLIERS Open Source Projects

You’ve got ... PARTS Open Source Components & Warehouses

You’ve got ...

MANUFACTURERS Software Development Teams

You’ve got ...

FINISHED GOODS Software Applications

SUPPLIERS: OPEN SOURCE PROJECTS

A massive community of open source projects represents the external supplier base that feeds software supply chains. But unlike physical parts being consumed within traditional supply chains, once a software component is made available, it is never taken out of circulation. This means developers inadvertently can continue to use the old, outdated parts even when newer, better versions are available.

Open source means open access. It is community driven and community supported. Open source is absolutely essential for today’s development processes. The increasing reliance on open source components has paralleled massive growth in component contributions from open source and third-party projects across all development languages. Where only a handful of open source projects were active in the early 1990’s, OpenHub. net is currently tracking over 668,000 open source projects and over 3. 7 million open source contributors². The massive demand for open source components within software supply chains is being fed by this growing community of contributors and projects.

One significant difference between traditional suppliers and open source suppliers is that, in the open source world, components never reach their “end of life” or get retired. Once released into public repositories, outdated components continue to be available to software supply chains.

Module Counts (Components)

development teams. 9

Image 3: Growth in new modules created by open source projects, 2014-2015. Source: http://www.modulecounts.com

SUPPLIERS Open Source

Projects

DID YOU KNOW?

Nearly 1,000 new or updated components are added to the Central Repository 5 every day.

On average, components are updated 3.5 times per year. There is no way to inform 9 development teams.

Mean-time-to- repair a security vulnerability in component dependencies is 10 390 days.

Public Repositories (The Warehouses)

In 2014, public repositories—which serve as “warehouses” at the heart of software supply chains—handled billions of download requests serving a global developer population of more 3 than 11 million.

Open source projects upload their open source components to a variety of warehouses where they can be freely shared by millions of developers anywhere, any time. These warehouses are commonly known in the software supply chain as public repositories such as the Central Repository, npmjs. org, rubygems. org, pypi. python. org, and NuGet. org. Here is a snapshot of recent use of public repositories:

Source: Each public repository provides statistics on activity levels for their communities.

The never-ending supplier ecosystem of components and versions

As mentioned earlier, the Central Repository is the source of record for Java and related open source software components. This repository is the most heavily used of its kind handling 17. 2 billion requests in 2014 alone⁴ and, as such, offers the widest-angle lens into the open source ecosystem and the software supply chains it serves. We have analyzed data from the Central Repository to help provide a better understanding of the software supply chain throughout this report.

Deeper analysis of the open source projects within the Central Repository showed more than 105,000 open source projects (categorized by Group-Artifact

• GAs) are housed there⁵. When considering multiple versions of each component (categorized by

Group-Artifact-Versions - GAVs), there were 834,399 6 components at the end of calendar year 2014. As of June 2015, the number of component versions 7 exceeded 9⁷4,000 and is well on its way to or above 1 million by the time you read this report. In the context of software supply chains, these projects represent the total number of suppliers while GAVs represent the total catalog of unique parts available.

Image 4: The growth of download requests from the Central Repository have grown 30-fold since 2007. Source: Sonatype.

Analysis of download requests from the Central 8 Repository reveals 106,0⁸7 organizations requested components in 2014. Automated requests can originate from a variety of sources, including popular build and design tools (Aether, Ant, Ivy, Leiningen,

Gradle, Maven, Eclipse), repository managers (Archiva, Artifactory, Nexus), and orchestration platforms (Puppet, Chef, Rundeck).

While the global base of open source projects has fed software supply chains for many years, they’re not the only suppliers. As development organizations look to push the envelope on speed, they are looking for new sources for software, infrastructure, and services. For example, developers are looking for new ways to create Docker images and share them with other developers. For this purpose, Docker, Inc. runs Docker Hub, allowing companies to search and pull images as needed. Organizations like CoreOS Enterprise Registry provide a similar service.

As popularity of reusable containerized images, microservices, binaries, and code continues to grow, the supplier ecosystem will continue to expand to support this need.

Lack of meaningful communications channels

Unlike a traditional supply chain where the parts supplier and manufacturer have a clear relationship and communications channel, in a software supply chain that communication channel is not only broken, for many it simply doesn’t exist.

While a handful of projects release new component versions weekly, on average a project will release new versions 3. 5 times a year⁹. These releases might provide new functionality, improve performance, fix bugs, or occasionally patch newly found security vulnerabilities (which are generally slower to be remediated). The lack of a communications channel, coupled with the fact that a component is never taken out of circulation, makes the chances even greater that development teams will unknowingly use outdated, defective parts.

The lack of a communications channel makes the chances even greater that development teams will uknowingly use outdated, defective parts.

Choosing the Best Suppliers (Sourcing)

There is a robust and active ecosystem of suppliers providing a steady stream of innovative com- ponents to feed the demands of software supply chains. However, like in traditional manufactur- ing, not all suppliers deliver parts of comparable quality and integrity. Research shows that some open source projects use restrictive licenses and vulnerable sub-components, and some projects are far more diligent at updating the overall quality of their components.

Choosing an open source project supplier should be considered an important strategic decision, because changing a supplier is far more effort than swapping out a specific component version. Like tradition-

al suppliers, open source projects have good and bad practices impacting the overall quality of their component parts. Where traditional manufacturing supply chains intentionally select specific parts from

approved suppliers, software supply chains rely on an unchecked variety of supply. Development teams choose whatever technology is deemed appropriate, as well as whatever version might be in vogue at the time of selection, introducing a significant degree of unnecessary complexity—and sometimes risk.

Hidden component defects

Not unlike traditional finished goods, a single part may rely on many other parts to function properly. In software development, components are like molecules, not atoms. A single component may rely on hundreds of other sub-components known as dependent components. A direct dependency (aka 1-hop) is a component directly tied to the core component. As shown in image 5, there are multiple layers of dependencies. This complex, inter-related web of components makes it even more difficult for some open source projects to manage vulnerabilities in their own software, even despite best efforts to produce quality code. As a result, neither the open source project—or the development teams who consume them—are aware of known vulnerabilities in dependent components.

Mean-time-to-repair (M-T-T-R)

Image 5: A single component may rely on hundreds of other sub-components known as dependent components.

Overall, defect rates between open and closed source projects have been well documented and are believed to be comparable. But until 2014, little quantitative analysis had been available on the mean-time-to-repair (MTTR) defects within open source projects. In order to shed new quantitative light on this discussion, Sonatype initiated an analysis of open source project code bases hosted within the Central Repository.

Although the average open source project may release new component versions 3. 5 times a year, it does not necessarily mean they are patching known security-related issues. In fact, in their seminal USENIX article Almost Too Big To Fail, Dan Geer and Josh Corman, shared findings from the Sonatype analysis. “An early analysis of open source projects with already identified vulnerable dependencies revealed some troubling behavior. Direct (aka ‘1-hop’) vulnerable component dependencies were only remediated 41% of the time. Put differently, more than half (59%) of the vulnerable base components remain unrepaired. Folding multiple components into your projects means inheriting not just the components’ functionality but also their (largely unrepaired) flaws. For the 41% that were fixed at all, the MTTR was 390 days (median 265 days). Filtering for just Common Vulnerability Scoring System (CVSS) level 10 vulnerabilities brought the mean of this subset down to 224 days. And this is just for 1-hop dependencies—there is as yet no mechanism to cause remediated flaws to flow automatically through the dependency graph, and 10 there may never be. ”

Hidden license complexity

Open source license risks are well known across the development community but are not always well understood. This is a key factor when choosing an open source supplier and component. According to long-time hosting provider SherWeb: “Taking what’s already been created, adding to it, improving upon it, and in turn sharing it with a worldwide community is what has allowed open source innovations to become ‘arguably the single most influential body of software around the world. ’ But it has also left

Open source is everywhere, from mobile phones to medical devices to supercomputers to home appliances, and as such patent lawyers have never been busier.

many open to attacks from those who would claim the process by which they invented software is theirs and theirs alone. Open source is everywhere, from mobile phones to medical devices to supercomputers

to home appliances, and as such patent lawyers have 11 never been busier. ”

Sonatype’s analysis of the components in the Central Repository revealed that 34% included restrictive GPL 12 licenses. Image 6 developed by Ritambhara Agrawal of Intelligere provides a high-level description of the different licenses often assigned to open source and third-party software components.

Many lawsuits have arisen from the improper use of open source software, including well-known cases like BusyBox vs. Monsoon, Cisco vs. FSF, and Oracle vs. Google. While an improper license will not impact the performance of software, it can lead to enormous headaches, legal fees, and unplanned rework to remove the undesirable component.

Free Open Source Software (FOSS) Licenses

Image 6: There are a wide variety of open source licenses, each with a different set of obligations for the user. Source: Brian Fitzgerald, Sonatype, Inc.

PARTS: OPEN SOURCE COMPONENTS AND LOCAL WAREHOUSES

Software components age more like milk than wine, so compo- nents that were once thought to be reliable can be discovered to be vulnerable in the future. Most troubling, organizations un- wittingly continue to use known vulnerable components at an alarming rate. Analysis shows that 51,000 of the components in the Central Repository have known security or license concerns and that 6.2% of all components downloaded from Central last 13 year included known vulnerabilities.

Due to a lack of adequate visibility, tools and processes, most development organizations are unaware of the known vulnerabilities or license risks in the open source components downloaded from the public repositories. There is a generally accepted notion that “with many eyeballs, all bugs are shallow” in open source components. It is true that broad open source adoption means that many organizations are—in effect—testing the components, however as noted previously:

1. Once a component is shared in a public repository, it stays there forever even after many newer, safer versions have been introduced.
   

2. A once safe component may be found to be vulnerable at any time.
   

3. Components often depend on other components in order to function, much like an engine needs multiple other parts. If a vulnerability is found in a component dependency, it is generally very difficult
   

for either the supplier (open source project) or the development teams to know about it—or track it down and fix it.

4. There is clear evidence that known vulnerable or defective components stored in public warehouses are downloaded by unknowing
   

development teams, and end up in our software largely unnoticed.

How can known vulnerabilities be largely unnoticed? There is a fundamental lack of automation to put essential, relevant information in the right hands, at the right time, and at the right place.

PARTS Open Source

Components & Warehouses

DID YOU KNOW?

There are nearly 1 million unique components in the Central Repository alone.

51,000 components in the Central Repository have a known security vulnerability.

283,000 components in the Central Repository have known restrictive licenses.

Source: See Appendix, Figure 1.

Image 7: Defective component downloads by large financial services and technology firms in 2014. Refer to Figure 6 in the appendix for more details.

Developers simply don’t have the time and are not incented to manually research each component. Other groups like open source review boards, legal and compliance teams, or security professionals are sometimes called in to support these efforts. But very few have the ability to track the current vulnerability or license status of components being used at the velocity in which they are being consumed.

The rapidly disseminated, global alert surrounding the OpenSSL Heartbleed vulnerability was an anomaly. While Heartbleed received international notoriety within a matter of days, there are an average of 50

While Heartbleed received inter- national notoriety within a matter of days, there are an average of 50 security vulnerabilities an- nounced in software every day that do not receive due attention.

security vulnerabilities found in open source components every day¹⁴ that do not receive due attention.755,437 downloads in 2014, Authors Geer and Corman also discuss this topic in their USENIX article¹⁵: “The ‘Legion of the Bouncy Castle Java Cryptography APIs’ had a CVSS worst-

case scenario fixed in April of 2008—more than six years ago. While CVE-2007-6721 is a severe security flaw in a security-sensitive project, nevertheless the unrepaired, vulnerable version was requested from 16 Central Repository 42,124 times in 2014.

Image 8: Organizations continue to use known vulnerable com- ponents years after alerts have been issued. Source: Sonatype.

Geer and Corman continued, “Similar (disappointing) consumption patterns exist for Struts. Outside of CVE- 2013-2251 compromised organizations, still vulnerable versions of Struts 2 continue to remain popular. Worse, Struts version 1-related artifacts still had ¹⁷ despite its April 5, 2013 public warning. In other words, finding and fixing serious flaws in open source does not mean that the repaired versions are the ones that are used. ”

Placing these comments in the context of traditional supply chains, imagine the impact of sourcing parts with known defects. Can you imagine an automaker that sourced defective Takata airbags for use in a

2015 model? Or a laptop designer that sourced batteries that are known to catch fire? Or a pharmaceutical company that uses compounds that are known to cause birth defects?

Industry Spotlight:

Current Practices >>

One large company in the entertainment industry established a team to monitor and approve open source components being requested by thousands of in-house developers. However, an analysis of their download traffic from the Central Repository revealed that 97% of components sourced from public repositories were outside of the purview of their open source review board. This scenario is quite common. While component consumption is high and continues to grow, full visibility and control are extremely rare.

Best Practices >>

To help defend the U. S. government cyber infrastructure, and to help the Department of Homeland Security and other agencies carry out their cyber defense mandate, U. S. Congressional Representatives Ed Royce (R-CA) and Lynn Jenkins (R-KS) introduced the “Cyber Supply Chain Management and Transparency Act of 2014. ” The proposed cyber legislation aims to ensure that any organization selling software, firmware or products to the federal government is properly managing its software supply chain. The legislation recommended a focus on three supply chain practices:

1. Any software, hardware, or firmware sold to a procuring entity must provide a Bill of Materials of third party
   

and open source components, including their versions.

2. Any software, hardware, or firmware cannot use known vulnerable components for which a less vulnerable
   

component is available (without a written and compelling justification accepted by procuring entity).

3. Any software, hardware, or firmware must be patchable/updateable (within a reasonable timeframe)—as
   

new vulnerabilities will inevitably be revealed.

Repository Management (Local Warehouses)

Repository managers are a fundamental first step toward software supply chain automation, fo- cusing primarily on management of component “parts” within an organization. In this sense, they serve as local warehouses providing development teams with more efficient and controlled com- ponent access enterprise-wide. They are used to streamline the acquisition, consumption, sharing and deployment of components downloaded from public repositories as well as other internal build components and artifacts.

Some software development organizations are beginning to automate their supply chains, establish better best practices, and measure benchmarks. For example, there are more than 60,000 repository manager installations, such as Sonatype Nexus, Apache Archiva or JFrog Artifactory. These repositories act as local warehouses to host open source and proprietary parts to improve build performance and shorten supply chain cycles.

While many development organizations have adopted build tools like Maven, Ant, Ivy, and Gradle, most have yet to fully employ a repository manager, both to proxy remote repositories and to manage and distribute software components.

In a software development shop, any number of these tools or developers can be simultaneously downloading components at-will, which helps explain the practices that lead to multiple versions of the same component being downloaded.

In an ideal software supply chain configuration, build tools would interact with a repository manager to search for binary software components and retrieve

Image 11: Growth in repository manager installations 2013-2015. Source: Sonatype’s Central Repository

software components on-demand. By caching and hosting components closer to the developers and tools that consume them, builds are faster, more efficient, and more reliable.

However, in most organizations build tools point to and retrieve components from public repositories directly (e. g., The Central Repository, NuGet Gallery, RubyGems. org). When analyzing the originating sources of downloads from the Central Repository, we see 95. 24% of the requests coming from a variety of design, build, integration, and orchestration tools. The remaining

18 4. 76% originate from repository managers.

In a software development shop, any number of these tools or developers can be simultaneously downloading components at-will, which helps explain the practices that lead to multiple defects and versions being downloaded.

Shortening supply chains,

speeding up development

If the software supply chain were fully optimized, we might expect to see the percentages in Image 13 flip.month. These percentages tell us that developers today are inefficiently sourcing virtually all of their components from distant central warehouses rather than targeting a source closer to home. Not only do long-distance requests take longer to fulfill, but they also add unnecessary bandwidth costs to operations.

cumulative effect across hundreds or thousands of developers can add weeks or months to build times to development teams who are already pressured to deliver faster.

For example, downtime can become a big factor for development teams sourcing components directly from a public repository. While mature repositories like the Central Repository have high availability rates, some of the less mature component repositories experience minutes or hours of downtime each When these outages occur, all development relying directly on the repositories grinds to a halt. The frustration experienced by these developers is evident when social media channels like Twitter light up with developers in frustrated unison, calling “[XYZ] repository is down. Fix it now!”

Once cached in their repository manager, developers

Image 13: Analysis of component distribution and sourcing reveals a significant level of inefficient behavior.

As a simple analogy, why would everyone in your neighborhood bypass the local grocery store and instead travel to a distant dairy farm in order to get their milk?

The impact of poor component consumption practices

While individual developers may not recognize the impact of their independent sourcing methods, the

and their development tools can retrieve the component locally, as many times as necessary across the organization regardless of the number of applications needing it. Repository managers support the concept of download-onceuse-many-times that improves sourcing practices and therefore are a foundational step in a broader trend towards improving performance and control of components across the software supply chain.

Why would everyone in your neighborhood bypass the local grocery store and instead travel to a distant dairy farm in order to get their milk?

The evolving role of repository managers in the software supply chain

To support continuous delivery, many organizations prefer to keep everything required to deploy or re-create an application’s binaries—and the environments in which they run—in a well-known, stable, and easily accessible location. Where for many years, developers have stored binary artifacts in their repository managers, they are now expanding its role to include other artifacts created and used across the continuous delivery pipeline.

Continuous Delivery Ltd. ’s Dave Farley shared an example (see Image 13) of the artifact repository (a. k. a., re-

Repository managers are now used as the internal “parts warehouse” for tests, database scripts, build and deployment scripts, virtual machines, containers, documentation, libraries, configuration files for your application, and so on. The idea is that at all times a project has a system of record for all binaries, environments, and executable deliverables that are known to be safe for development and deployment.

Repository managers are now used as a the internal “parts warehouse” for tests, database scripts, build and deployment scripts, virtual machines, containers, documentation, libraries, configuration files for your application, and so on.

pository manager) at the heart of a continuous deployment pipeline. In his example, the artifact repository (a. k. a., repository manager) acts as the central system of record for all inputs and outputs of the software supply chain’s production line.

Synchronizing teams across the software supply chain

Most anyone who has done maintenance programming has had the experience of not being able to recreate a defect because a change in one of the

Image 13: Artifact repositories play a central role in deployment pipeline. Source: Dave Farley, Continuous Delivery Ltd. 2015. Slides at: http://bit.ly/1C9y7X8

tools makes the original binary irreproducible. The discipline of using a repository manager also ensures that everyone is using the same set of documents and tools in development. This approach reduces the possibility that team members in other locations or overseas are using different requirements, building on new versions or a newer version of the compiler, etc.

In most organizations, more than one repository manager is used. The repository managers then are synchronized to ensure all development teams have access to the same artifacts for development and production. When implemented correctly, everyone on the team is essentially drawing from the same warehouse.

Using version control for all production artifacts

In leading DevOps and continuous delivery practices, it’s easy to recreate environments for testing and troubleshooting. Absolutely all artifacts used to create production environments and the application that run in them are version controlled. The ability to get changes into production repeatedly in a reliable, low-risk way depends on the comprehensive use of version control.

Similar to practices in other highly-tuned supply chains, leading software development teams also pursue strict version controls and traceability for all of their components. The breadth of version consumption is limited and if any quality defects are later identified in components that were used, they can be found in minutes versus weeks.

Industry Spotlight:

Current Practices >>

Analysis of some of the world’s largest financial services and technology firms demonstrates that use of repository managers alone does not improve the quality of components made available to developers. While companies downloaded an average of 29,697 components to their repository managers, on average, in these 19 large organizations 7% of all component requests had known defects. While many companies pursue “Golden Repository” strategies, where attempts are made to keep only approved components in their repository managers, we now have quantifiable evidence to show the approach does not work without additional rigor.

Best Practices >>

A global investment banking titan had empowered their developers to consume new components at-will, through any available channel. When they learned that thousands of in-house developers bypassed their repository manager to download over 2. 6 million components, they recognized the inefficiency of their sourcing practice as well as the potential for voluntarily inheriting quality, license, and security issues. The bank moved quickly. They required developers to first query local repository managers for components and, within one year, the bank reduced Central Repository downloads from 2. 6 million to 95,000 components. By reducing component downloads, the company eliminated complexity and inefficiency from their software supply chain which lead to an estimated yearly savings of 30 days of build time.

MANUFACTURERS: ASSEMBLED SOFTWARE DEVELOPMENT

Most development teams strive for ever-increasing speed and throughput. Yet the software assembly process remains rife with inefficiencies, largely due to a lack of enforceable policies and guardrails to help developers make better, safer decisions. This is not a “people problem.” This is an automation problem.

Very few CIOs, software development executives, enterprise architects, and especially personnel residing outside of the IT organization realize the extent of their organization’s reliance on the components and the supply chain that serves them. While legal, security, audit, open source review boards, and other functional organizations have attempted to detail and track consumption behaviors, they often fall short of gaining full visibility.

Time pressures can trump quality

Developers take great pride in their innovations, however time pressures often trump quality especially when time-consuming manual research is required to properly analyze component suitability. According to the results of of a 2014 Open Source and Application Security 20 survey of over 3,300 developers and IT staff.

• 43% of organizations don’t have development policies that address open source and third-party component usage across versions, age, licenses, and security issues.

Source: Sonatype 2014 survey²⁰

MANUFACTURERS Software

Development Teams

DID YOU KNOW?

1 in every 16 component downloads included a known security 4 vulnerability.

43% don’t have policies to manage 20 component quality.

component quality. 20

75% of those with policies don’t enforce 20 them.

31% have had or suspect a breach in an open source 20 component.

Source: 2014 Sonatype Open Source Development and Application Security Survey.

• Of the organizations with policies in place, 75% of participants indicated that the policies were not enforced.

Automating acceptability

Organizations wishing to improve the quality and integrity of components being consumed must rely on

Source: Sonatype 2014 survey²⁰

• In organizations with more than 500 developers, the situation was somewhat better as 73% of these organizations had a policy in place and 49% stated that use of open source and third-party 21 components was strictly enforced.

automation. In light of the volume and velocity of component consumption we have already noted, clutching to current manual processes cannot enable an organization to reach new levels of productivity and competitiveness.

DevOps and Continuous Delivery teams should not just go through a checklist of acceptability, but also actually define the attributes that make a software component acceptable for use. Automated policies can then enforce things like “don’t allow anything into my billing process if it has a severe

security defect,” or “don’t allow anything into my customer-facing applications that leverage the fair use GPL licenses. ” Humans need only define the policies, then use automation for enforcement. Then humans can manage the occasional exception to the automated process.

Very few CIOs, software development executives, enterprise architects, and especially personnel residing outside of the IT organization realize the extent of their organization’s reliance on the components and the supply chain that serves them.

Technical Debt: Assembly Line Inefficiencies

In one large financial services firm, developers had downloaded 81 of the 85 versions of the Spring Core framework in 2014. This not only means the investment bank had access to a wide variety of outdated versions, but that their developers potentially and unknowingly added unnecessary technical bloat and maintainability challenges to their portfolio.

Within the top 100 most popular components, we saw that 27 versions of a single component were downloaded by an average organization in 2014 ²² alone. By consuming the poor quality, risky, or defective components across their supply chains, organizations were electively building in technical and security debt. And since few organizations keep track of which components were used in each application, the time, effort, and cost of identifying and remediating those risks is substantial.

Imagine if a company manufacturing insulin pumps was able to freely select any one of 27 different pumping mechanisms. The impact is far reaching: it impacts the assembly line, customer service, and quality control. The same is true for software.

The cost of context switching

As development teams move toward continuous delivery and agile practices, they are even more likely to work on multiple projects in a given year. When you

Image 10: The impact of context switching on development teams. Source: Dominica deGrandis, DevOps Days Austin 2015

are not only switching from one project to the next, but also switching between multiple component versions, the impact of context switching is even greater. As image 10 shows, context switching is proven to not only impact development speed, but also quality and job satisfaction.

Industry Spotlight:

Current Practices >>

While assisting a federal government organization in analyzing components in a large web application that helps service millions of citizens, an auditor discovered 11 different logging framework components in use. This behavior is similar to an automobile manufacturer using four different versions of door locks from four 23 unique suppliers in one vehicle.

Best Practices >>

A global Internet search and advertising giant uses tens of thousands of open source and third-party components across its developer community. To limit the complexity of its operations, developers are free to use any high-quality software component, but the company restricts them to using only one of the latest, most stable versions. By limiting the number of versions used, developers can easily shift between projects without needing to learn the nuances of an alternative version. It also takes the company less time to iden- **2015 State of the Software Supply Chain Report: Hidden Speed Bumps on the Road to “Continuous”**24 tify and repair defects.

FINISHED GOODS: SOFTWARE APPLICATIONS

In the traditional manufacturing supply chain, product quality and safety are enhanced through diligent tracking of the suppliers and parts used in each product. However, in the software supply chain, where an average of 106 components²⁵ comprise 80-90% of the total application, few organizations have visibility into what components were used and where.

The Volume of Elective Re-work and Risk

Known defective components lead to quality and integrity issues within applications. While developers save tremendous amounts of time by electively sourcing software components from outside their organizations, they often don’t have time to check those component versions against known vulnerability databases, internal open source policies, and other sources of quality information. Earlier in this report, we showed 6. 2% of downloads from the Central Repository—at the front of the software supply chain—were components that included known vulnerabilities.

Image 9: Components and known security or license risks in an average application

Analysis of over 1,500 applications reveals that by the time the applications are developed and released at the end of the software supply chain, a typi-

cal application has 24 known severe or critical security vulnerabilities and 9 26 restrictive licenses.

Likewise, in a security analysis across 5,300 applications, Veracode also found and confirmed that an average application has 24 known security 27 vulnerabilities associated with open source and third-party components.

Where most organizations have relied on manual reviews of open source components used in applications, these practices are proving insufficient. Based on the volume and velocity of open source and third-party software components being consumed, it is impossible to check everything manually. It is simply too expensive and too slow—especially given the sub-components or dependencies which are less obvious.

FINISHED GOODS Software

Applications DID YOU KNOW?

23% of the components in the average application have critical or severe known 26 vulnerabilities.

There are 9 restrictive licenses per application, critical or 26 severe.

63% of organizations keep an incomplete software Bill of 28 Materials.

Software Bill of Materials

Unlike “bill of materials software” which is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a “software bill of materials“ (BOM) is an inventory of the third party and open source components used to build an application.

As noted in Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part 29 of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. real-time component defect data to indicate which If defects are later found in a specific part, the BOM makes it easy to locate affected products.

“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to

3 date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in 30,31 managing risk. ”

A software bill of materials not only inventories what is used, but in some cases it also syncs with components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.

Quality Controls: OWASP, PCI, FS-ISAC, U.S. Congress

An executive from a well-known U.S. federal agency once said: “There is no building code for building code.” While that is not 100% true, it does accurately describe the component-based portion of modern software development. Among the many benefits of supply chain automation in other industries, quality controls have made products safer and more reliable. However, in the software supply chain, the lack of controls can, in some cases, result in critical security defects that may put consumers at risk of losing their credit card data or healthcare history to the highest bidder in the black market. In the case of defective open source components in medical devices or cars, the risks can be even greater.

Recognizing the growing and ongoing threat posed by the use of defective component parts, organizations representing IT, financial services, healthcare, and application development have proposed new guidelines for ensuring the highest-quality parts are selected, used, and traceable across application portfolios. These include:

The Open Web Application Security Project (OWASP): In 2013, OWASP updated its top ten list of application security threats to include A9, which advises against “using components with known vulnerabilities. ” The full description from OWASP states: “Components, such as libraries, frameworks, and other software modules, almost always run with full privileges. If a vulnerable component is exploited, such an attack can facilitate serious data loss or server

takeover. Applications using components with known vulnerabilities may undermine application defenses 32 and enable a range of possible attacks and impacts. ”

Payment Card Industry (PCI): PCI standards help ensure that banks, financial services firms, and merchants protect their customers’ credit card data. PCI guidelines were quickly updated following the OWASP A9 guidelines addressing known vulnerabilities.

Financial Services Information Sharing and Analy- sis Center (FS-ISAC): The FS-ISAC started the Product & Services Committee to identify appropriate security control types for third-party service and product providers. This effort is due to the fact that the application represents the “new perimeter. ” The working group references Gartner research that states, “Since enterprises are getting better at defending perimeters, attackers are targeting IT supply chains. ” The FS-ISAC report also stated, “Recent breach reports such as Verizon’s Data Breach Investigations Report underscore the vulnerability of the application layer, including third-party software. This new perimeter of 33 third party software must be addressed. ”

the new supply chain reality: vBSIMM process maturity assessment, binary static analysis, and policy management and enforcement for consumption of open source libraries and components.

The FS-ISAC committee addressed three suggested control types that should be implemented based on

U.S. Congress: To help defend the U. S. government cyber infrastructure and help the Department of Homeland Security and other agencies carry out their cyber defense mandate, U. S. Congressional Representatives Ed Royce (R-CA) and Lynn Jenkins (R-KS) introduced the “Cyber Supply Chain Management and Transparency Act of 2014. ” The proposed cyber legislation aims to ensure that any organization selling software, firmware, or products to the federal government is properly managing its software supply chain. The legislation recommended a focus on three supply chain practices: (1) any software, hardware, or firmware sold to a procuring entity must provide a bill of materials of third-party and open source components, including their versions; (2) any software, hardware, or firmware cannot use known vulnerable components for which a less vulnerable component is available (without a written and compelling justification accepted by procuring entity); and (3) any software, hardware, or firmware must be patchable/ updateable (within a reasonable timeframe)—as new 34 vulnerabilities will inevitably be revealed.

LESSONS LEARNED FROM TRADITIONAL

MANUFACTURING SUPPLY CHAINS

What if manufacturers built cars using the same processes currently used to build software? They could choose any part from any supplier they wanted. They could choose these parts without visibility into quality, or which parts or versions their colleagues were using. Every car would likely have a unique set of parts and a wide variety of known defects already built in as it rolled off the assembly line. Months or years later, when a defect was widely publicized, they wouldn’t know if they used that part and, if so, where.

Leading organizations like Toyota learned they could sustain a competitive advantage by following three basic principles—use fewer and better suppliers, use higher quality parts, and track what is used and where.

Software is not the first industry to face supply chain challenges. From automotive to energy, healthcare to defense, mature industries have automated their supply chain to produce goods quickly and efficiently with high quality and minimized risk. The benefits include improved performance, profitability, transparency, and sustainable competitive advantages.

Toyota has reduced complexity by using only 125 plant suppliers for the Prius. General Motors has 800 for the Volt. General Motors produces 54% of the content of their vehicles while Toyota produces 27%.figure 5 for more information.

General Motors has 20x the suppliers and yet they produce half of the content of their vehicles. Therefore, it’s no surprise that the Volt cost nearly $35,000 and a Prius less than $25,000. At the time of this research, Toyota sold 20,000 Prius units a month and 35 General Motors sold 1,700 Chevy Volts.

Interestingly, the software development practices of the majority of organizations today tend to have more in common with General Motors than you might think. And in fact, it is far more troubling because—in a typical software supply chain—developers don’t have visibility into component quality, yet each developer can choose any component they want. Freedom of choice combined with lack of quality information is a toxic combination. (See appendix, )

Image 2: Supplier complexity impacts cost and quality, two important factors for competitive differentiation. The same is true of software. Source: Toyota Supply Chain Management: A Strategic Approach to Toyota’s Renowned System, by Ananth Iyer and Sridhar Seshadri

AUTOMATION: HOW TO IMPROVE SOFTWARE SUPPLY CHAINS

Unnecessary—and largely hidden—complexity permeates nearly all software development to- day. If it continues along the same path, these speed bumps will block further increases in produc- tivity and the on-time, on-budget delivery of quality software.

The introduction of high-level languages, objectoriented programming, agile, continuous delivery, DevOps, and dependency management are all ways in which software development practices have responded to problems with scalability, complexity, and the desire to unlock the next level of efficiency.

Software supply chain complexities, coupled with the volume and velocity of open source and third-party components flowing through them, have created such a requirement. Soon, the industry at large will begin to demand solutions in response. But the answer is here today: automation.

Mother Teresa said, “If I look at the mass, I will never act. If I look at the one, I will. ” Looking at the massive volume and velocity of open source and other artifact consumption can be discouraging and overwhelming. However, the volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.

Automation in areas of testing, build, and deployment has provided significant performance benefits. Likewise, investments in software supply chain automation have shown markedly improved efficiency and controlled risk, as the best practices in this report illustrate. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality.

Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity—with gains on par or even greater than possible with agile, Lean, and DevOps.

To begin improving your software supply chain, consider these next steps:

1. Create a software bill of materials for one appli- cation: Visibility into one application can help you better understand your current component usage. A number of free and paid services are available to help you create a software bill of materials within a few minutes. The bill of materials will help you to identify the unique component parts used within your application and the suppliers who contributed them. These reports list all components used, and several services also identify component age, popularity, version numbers, licenses, and known vulnerabilities.

2. Take inventory within one of your local ware- houses: Repository managers (a. k. a., artifact repositories) are commonly used by software development teams. Components residing in repository managers can easily be reused across software development. If outdated or defective components reside in the repository manager, they can easily make their way into multiple applications. Start by identifying a repository manager in use within your organization and creating an inventory of the parts cached there. Some repository managers automate this inventory reporting process for you, listing components, versions, licenses, and security vulnerabilities.

3. Design approval processes to be frictionless, scalable, and automated: Manual reviews of components and suppliers cannot keep pace with the current volume and velocity of consumption. Your organization must not only define your policies for supplier and part selection but also find practical ways to enforce them without slowing down development or inadvertently encouraging workarounds. Policies must be agile enough to keep pace with modern development. Strive to automate policy enforcement and minimize drag on developers.

4. Enable developer decision support: Developers are primary consumers of components in software supply chains. They initiate every component request. Help developers by automating the availability of information on component versions, age, popularity, licenses, and known vulnerabilities within their existing development tools so it is By selecting the highest-quality components from

the highest-quality suppliers early, you will improve developer productivity and reduce costs.

5. Ensure visibility and traceability throughout the software lifecycle: Components can enter the software development lifecycle at many different places. Requests are initiated by developers and the tools they use regularly to build, integrate, and release applications. Since component selection is not a point-in-time event, continuous monitoring should be used to alert you when new and unchecked components have entered the supply chain. Alerts can also tell you when components used are out-of-date or when new vulnerabilities have been discovered. Continuous monitoring will also allow for component traceability, improving overall mean-time-to-repair defects.

Organizations that have taken control of their software supply chains have seen tremendous gains in developer productivity, improved quality, and lower risk. This report has highlighted a number of industry-leading practices. As your organization embarks on its journey to supply chain automation, you can use these easy to pick the best components from the start. examples to establish new performance, quality, and productivity benchmarks for your organization.

ABOUT SONATYPE

Sonatype helps organizations build better software, even faster. Like a traditional supply chain, software applications are built by assembling open source and third party components streaming in from a wide variety of public and internal sources. While re-use is far faster than custom code, the flow of components into and through an organization remains complex and inefficient. Sonatype’s Nexus platform applies proven supply chain principles to increase speed, efficiency and quality by optimizing the component supply chain. Sonatype has been on the forefront of creating tools to to improve developer efficiency and quality since the inception of the Central Repository and Apache Maven in 2001, and the company continues to serve as the steward of the Central Repository serving 17. 2 Billion component download requests in 2014 alone. Sonatype is privately held with investments from New Enterprise Associates (NEA), Accel Partners, Bay Partners, Hummer Winblad Venture Partners and Morgenthaler Ventures. Visit: www. sonatype. com

APPENDIX

Figure 1: The Volume and Size of the Global Software Supply Chain based on data compiled from the Central Repository.

This chart provides visibility to the number of suppliers, parts, and warehouses used across the software supply chain, specific to Java open source components. The volume of download requests, including those with known vulnerabilities, has been noted in the chart. The chart represents data analyzed from the Central Repository (www.sonatype.org) from calendar year 2014.

Figure 2: Target Benchmarks for Software Supply Chain Best Practices - Quality Control

Sonatype research and analysis has revealed notable differences in quality control practices and policies across organizations. While the best practice would be to eliminate all use of known vulnerable or poor-quality components, Sonatype recognizes that as an end goal. Industry guidelines developed by OWASP, FS-ISAC, and PCI recommend not using components with known vulnerabilities in applications. While a number of organizations have made significant progress on this front, Sonatype decided to highlight some of the better examples of policies and practices, understanding that we have significant room to improve across all industries.

Figure 3: Target Benchmarks for Software Supply Chain Practices - Efficient Distribution

Sonatype research and analysis has revealed notable differences in the sourcing of open source components by development teams. These sourcing practices sometimes represent efficient distribution of components across the software supply chain by utilizing local warehouses (i.e., repository managers). Inefficient sourcing practices would bypass the local repository manager where the requested component was stored and go directly to the distant public repository (i.e., central warehouse) for retrieval of component parts.

Figure 4: Analysis of Components Used within Applications

This chart provides a summary of more than 1,500 applications analyzed by software developers using Sonatype’s Application Health Check (AHC), a community service offering available at no cost. The AHC details component popularity, age, license types, and known security vulnerabilities as part of providing a software bill of materials.

Figure 5: Multiple Versions of Parts Often Downloaded by the Largest Development Teams

Sonatype analyzed the 2014 component downloads of 29 large companies in the financial services and technology industry.
Those companies downloaded different volumes ranging from 10,000 to 900,000 components. Sonatype grouped the organizations in volume download bands to compare the number of orders, suppliers, and the variety of parts used across their software supply chains.

Figure 6: Volume of Defective Parts Used

Sonatype analyzed the 2014 component downloads of 29 large companies in the financial services and technology industry.
Those companies downloaded different volumes ranging from 10,000 to 900,000 components. We grouped the organizations in volume download bands to compare the number defects parts (i.e., components with known vulnerabilities) downloaded from the Central Repository. We also analyzed the age of CVE downloads.

Figure 7: Comparison of Impact of Supply Chain Complexity on Prius versus Volt

Other industries have reduced supply chain complexity by using the highest quality parts from fewer and better quality suppliers. One example highlighted in the book “Toyota Supply Chain Management” by Ananth Iyer and Sridhar Seshadri reveals the advantages Toyota’s Prius recognized over GM’s Chevy Volt.

Figure 8: Efficient Sourcing Practices By Manufacturers

Sonatype analyzed the 2014 component downloads of 29 large companies in the financial services and technology industry.
Those companies downloaded different volumes ranging from 10,000 to 900,000 components. We grouped the organizations in volume download bands to compare the volume of components being downloaded by repository managers. Component downloads by a repository manager represent a more efficient practice of sourcing by software development organizations.
Sonatype also analyzed the number of defective components being downloaded to these repository managers.

Footnotes:

1. Sonatype research including Application Health Checks and Open Source surveys, 2013 – 2014. 2. Openhub, https://www.openhub.net 3. IDC, http://www.idc.com/getdoc.jsp?containerId=prUS24529613 4. Sonatype analysis of the Central Repository for 2014 5. Sonatype analysis of the Central Repository for 2014. Statistics are updated daily at https://search.maven.org/#stats 6. Sonatype analysis of the Central Repository for 2014 7. Sonatype, https://search.maven.org/#stats 8. Sonatype analysis of the Central Repository for 2014 9. Sonatype analysis of the Central Repository for 2014 10. Almost Too Big to Fail, https://www.usenix.org/system/files/login/articles/15_geer_0.pdf 11. SherWeb, http://www.sherweb.com/blog/5-pivotal-open-source-lawsuits/ 12. Sonatype analysis of the Central Repository for 2014 13. Sonatype analysis of the Central Repository for 2014 14. National Vulnerability Database, https://nvd.nist.gov 15. Almost Too Big to Fail, https://www.usenix.org/system/files/login/articles/15_geer_0.pdf 16. Sonatype analysis of the Central Repository for 2014. The original article provided stats for calendar 2013. The stats have been updated in this report to reflect calendar 2014. 17. Sonatype analysis of the Central Repository for 2014. The original article provided stats for calendar 2013. The stats have been updated in this report to reflect calendar 2014. 18. Sonatype analysis of the Central Repository for 2014 19. Sonatype analysis of 29 large financial services and technology sector companies and their Central Repository activity, April 2014 – March 2015. 20. Sonatype, Open Source Development and Application Security Survey 2014, bit.ly/2014OSS_survey 21. Sonatype analysis of Open Source Development and Application Security Survey 2014 results. 22. Sonatype analysis of 29 large financial services and technology sector companies and their Central Repository activity, April 2014 – March 2015. 23. Rugged DevOps: Going Even Faster with Software Supply Chains, presentation by Josh Corman and Gene Kim, RSA Conference, April 2015. 24. Rugged DevOps: Going Even Faster with Software Supply Chains, presentation by Josh Corman and Gene Kim, RSA Conference, April 2015. 25. Sonatype, 2014 analysis of Application Health Check results. 26. Sonatype, 2014 analysis of Application Health Check results. 27. Veracode, http://www.veracode.com/open-source-and-third-party-components-embed-24-known-vulnerabilities-every-web-application-average 28. Sonatype, Open Source Development and Application Security Survey 2014, bit.ly/2014OSS_survey 29. Wikipedia 30. Wikipedia 31. Wikipedia 32. Open Web Application Security Project, https://www.owasp.org/index.php/Top_10_2013-A9-Using_Components_with_Known_Vulnerabilities 33. FS-ISAC, http://bit.ly/fsisac_knownvulnerability 34. U.S. Representative Royce, http://royce.house.gov/news/documentsingle.aspx?DocumentID=397589

35. Toyota Supply Chain Management: A Strategic Approach to Toyota’s Renowned System, by Ananth Iyer and Sridhar Seshadri