2020 of Statethe Software Supply Chain

The 6th Annual Report on Global Open Source Software Development

IN PARTNERSHIP WITH

sonatype

PRESENTED BY

Contents

Contents Introduction.............................................................4 CHAPTER 1 Open Season on Open Source...................... 5 Software Supply Chain Attacks: Past and Future............................................................... 6 Rise of Next-Gen Software Supply Chain Attacks (2015-2020)..........................................7 Speed Remains Critical When Responding to Legacy Software Supply Chain Attacks ..................................................10 CHAPTER 2 Open Source: Supply and Demand............12 JavaScript.........................................................................13 Java....................................................................................14 .NET....................................................................................14 DockerHub .....................................................................14 CHAPTER 3 Identifying Exemplary Open Source Suppliers....................................15 Researching the Best Performing OSS Projects............................................16 Finding Different Behavioral Groups.....................16 Exemplars.........................................................................16 Laggards........................................................................... 17 Cautious Teams.............................................................. 17 Projects with Updated Dependencies Are More Secure............................18

Guidance for Open Source Project Owners and Contributors...........................19 Guidance for Enterprise Development Teams....................................................19

CHAPTER 4 How High Performance Teams Manage Open Source Software Supply Chains................................ 20 Survey of Open Source Management Practices...............................................21 Comparing High Performers vs. Low Performers....................................................... 23 Comparing High Performers vs. Security First ........................................................... 23 Variables Most Impacting Performance and Risk Management....................24 Influencing Risk Management Outcomes............ 24 Influencing Productivity Outcomes........................26 Influencing Job Satisfaction...................................... 27 Guidance for Enterprise Development Teams...................................................27 Patterns Across OSS Component Updates: Easy, Difficult, and Planned...................28

Australia. ..........................................................................39 Variables Most Impacting

Software Supply Chains . ................................. 31 Laggards........................................................................... 17

OSS Components Make Up 90% of a Modern Application..................................33 21% of Enterprises Experienced Open Source Breaches.............................................34

The Changing OSS Landscape: Social Activism and Government Standards...................................35 Social Activism and Open Source Software......36 Governments Apply New Standards to Secure Software Supply Chains........................36 United States..................................................................36 United Kingdom............................................................ 38 Australia...........................................................................39

CHAPTER 5 The Trust and Integrity of Software Supply Chains.................................. 31 1 in 10 OSS Downloads Are Vulnerable ..............32 Enterprises Rely on Code From 3,500 Suppliers, But Quality Varies.......................33

High Performers detect and remediate OSS vulnerabilities 26x faster pg. 23

11% of components used in applications are known vulnerable pg. 34

47% of survey vulnerable pg. 34

High Performers are 51% more likely to create a software bill of materials (SBOM) pg. 23

requests expected

of all npm packages

rely on code with known

On average, there are 38 known OSS vulnerabilities per application

High Performers are 59% more likely to be using software composition analysis (SCA) tools pg. 23

NIST introduces new standards that call for SBOMs and OSS security checks pg. 38

430% YOY growth in cyber attacks

in cyber attacks targeting open source software projects pg. 6

pg. 16 Exemplary projects have 1.5x more frequent releases

Exemplary projects are 530x faster at updating dependencies

1 in 10 component

373,000 average enterprise downloads of OSS components per year pg. 33

average enterprise downloads that call for SBOMs

of OSS components per year and OSS security

High Performers are 28% more likely to enforce OSS governance in Continuous Integration (CI) pg. 23

Introduction

Digital innovation is the ultimate source of competitiveness and value creation for almost every type of business. As a result, three things are increasingly common among corporate software engineering teams and the 20 million software developers that work for them:

⊲ They seek faster innovation

⊲ They seek improved security

⊲ They utilize a massive volume of open source libraries

The universal desire for faster innovation demands efficient reuse of code, which in turn has led to a growing dependence on open source and thirdparty software libraries. These artifacts serve as reusable building blocks, which are fed into public repositories (npm, Maven Central, PyPI, NuGet Gallery, RubyGems, etc.) where they are freely borrowed by millions of developers in the pursuit of faster innovation. This is the definition of the modern software supply chain.

Now in its sixth year, Sonatype’s State of the Software Supply Chain Report continues to examine compelling and measurable practices of secure open source software development and delivery. For the second year in a row, we’ve collaborated with research partners Gene Kim from IT Revolution and Dr. Stephen Magill, CEO at MuseDev, to examine how high performing enterprise software development teams successfully balance their

performance and risk management practices while assembling applications with open source components.

The 2020 State of the Software Supply Chain Report blends a broad set of public and proprietary data, along with survey results from over 5,600 professional developers to reveal important findings, including:

⊲ 430% growth in next generation cyber attacks actively targeting open source software projects (Chapter 1)

⊲ 1.5 trillion open source component and container download requests in 2020 (Chapter 2)

Once again, the report summarizes the latest government and industry initiatives designed to protect software supply chains and strengthen the foundations of open source.

⊲ 530x faster mean time to update dependencies and 2.8x more commits for exemplary open source projects (Chapter 3)

⊲ 26x faster detection and remediation of open source vulnerabilities for high performance enterprise development teams (Chapter 4)

CHAPTER 1 Open Season

Open Season on Open Source

CHAPTER 1: OPEN SEASON ON OPEN SOURCE

In the past 12 months, the number of next gener- ation cyber attacks aimed at actively infiltrating open source increased 430%. The attacks are a uniquely efficient way for adversaries to gain leverage and scale by exploiting software supply chains.

Simply stated, members of the world’s open source community are facing a novel and rapidly expanding threat that has nothing to do with passive adversaries exploiting known vulnerabilities in the wild — and everything to do with aggressive attackers implanting malware directly into open source projects. To that end, it is important to distinguish between legacy supply chain exploits, and next-generation supply chain attacks.

Software Supply Chain

Attacks: Past and Future

Legacy software supply chain “exploits,” such as the now famous Struts incident at Equifax, prey on publicly disclosed open source vulnerabilities that are left unpatched in the wild. Conversely, next generation software supply chain “attacks” are far more sinister because bad actors are no longer waiting for public vulnerability disclosures. Instead, they are taking the initiative and actively injecting malicious code into open source projects that feed the global supply chain. By shifting their focus “upstream,” bad actors can infect a single component, which will then be distributed “down- stream” using legitimate software workflows and update mechanisms. Two high profile examples of 1 these modern upstream attacks are event-stream,

3. The ethos of open source is built on “shared

als, which creates a fertile environment whereby

which targeted the Copay cryptocurrency wallet in November 2018, and the recent Octopus Scanner Malware targeting the NetBeans open source IDE 2 in May 2020.

According to security researchers at the University of Bonn, SAP Labs France, and Fraunhofer FKIE, “From an attacker’s point of view, [large scale, public internet-based] package repositories represent a reliable and scalable malware distribution channel. Thus far, Node.js (npm) and Python (PyPI) repositories have been the primary targets of malicious packages, supposedly due to the fact that malicious code can 3 be easily triggered during package installation.”

FIGURE 1A Combined Reach of 100 Influential Maintainers Combined Reach of 100 SOURCE: MARKUS ZIMMERMANN AND CRISTIAN-ALEXANDRU STAICU, TU DARMSTADT; CAM TENNY, R2C; MICHAEL PRADEL, TU DARMSTADTInfluential Maintainers

Next-generation software supply chain attacks are possible for three reasons:

1. Open source projects rely on contributions from thousands of volunteer developers, and discriminating between community members with good or malicious intent is difficult, if not impossible.

2. Open source projects themselves typically incorporate hundreds — if not thousands — of dependencies from other open source projects, which may contain known vulnerabilities. While some open source projects demonstrate exemplary hygiene as measured by mean time to remediate (MTTR) and mean time to update (MTTU), many others do not (see Chapter 3). The sheer volume of open source in use and the massive number of dependencies makes it difficult to quickly evaluate the quality and security of every new version of a dependency.

Evolution of Package Reach for the Top 5 npm Packages Evolution of Package Reach for SOURCE: MARKUS ZIMMERMANN AND CRISTIAN-ALEXANDRU STAICU, TU DARMSTADT; the Top 5 npm Packages CAM TENNY, R2C; MICHAEL PRADEL, TU DARMSTADT

SOURCE 1A, 1B: Markus Zimmermann and Cristian-Alexandru Staicu, TU Darmstadt; Cam Tenny, r2c; Michael Pradel, TU Darmstadt

FIGURE 1B Evolution of Package Reach

CHAPTER 1: OPEN SEASON ON OPEN SOURCE

Furthermore, the researchers found that the

package reach of the top 5 packages was

between 134,774 and 166,086 other packages,

making them an extremely attractive target for

attackers (FIGURE 1B). 5

February 2015 to June 2019, 216 such attacks

rest-client, August 2019), releasing new versions of a project to a public repository (see bootstrap-sass ,

were hosted under individual developer accounts; (FIGURE 1C).

In 2019 Darmstadt University researchers found that a typical npm package contained an abnormally large number of dependencies — loading an average of 79 third-party packages from 39 different maintainers. The research team also found that 391 highly influential project contribu- tors affect more than 10,000 components through 4 their complex web of dependencies.

their complex web of dependencies. 4

If an adversary were to successfully identify entry points into projects supported by one of these 391 maintainers, they could dramatically widen the aperture and impact of their open source supply chain attacks. For example, the Darmstadt team said that adversaries gaining access to 20 popular npm maintainer accounts could deploy malicious code impacting more than half of the npm ecosys- tem (FIGURE 1A).

Rise of Next-Gen

Software Supply Chain

Attacks (2015-2020)

Exacerbating the risks even further, the Linux Foundation’s Core Infrastructure Initiative found that of the top 10 most-used software packages, seven were hosted under individual developer accounts; the researchers then questioned “what happens if one of these accounts is hacked? Would you, farther 6 down the software supply chain, even know?”

tions (see Octopus Scanner , May 2020).

Next generation cyber attacks actively targeting open source software projects have increased 430% since we published this report last year. From

The most common type of attack is Typosquatting, an indirect attack vector that preys on developers making otherwise innocent typos when searching for popular components. If a developer accidentally types “lodahs” when their intention is to source “lodash,” they might accidentally install a malicious component of a similar name (see Lodahs, November 2019).

When malicious code is deliberately and secretly injected upstream into open source projects, it is highly likely that no one knows the malware is there, except for the person that planted it. This approach allows adversaries to surreptitiously set traps upstream, and then carry out attacks downstream once the vulnerable code has moved through the supply chain and into the wild.

CHAPTER 1: OPEN SEASON ON OPEN SOURCE

attacker to compromise the affected system. 15

An abbreviated list of next-generation software supply chain attacks occurring from January 2019 – May 2020:

JANUARY 2019 ⊲ pytz3-dev

⊲ pytz3-dev The author of this PyPI package seems to have

The author of this PyPI package seems to have copied the ‘pytz’ package code and then added malicious code that finds the Discord application’s data folder on Windows machines and attempts to extract the Discord token from a SQLite database file. The package has been downloaded about 47 7 times per month.

⊲ smartsearchwp

Published in January 2019 and then yanked from the npm repository in June 2020, included malicious code 8 that provided a backdoor to support data exfiltration.

MARCH 2019 ⊲ simple-captcha2 0.2.3 and datgrid 1.0.6

⊲ simple-captcha2 0.2.3 and datgrid 1.0.6 As distributed on RubyGems.org, included a

As distributed on RubyGems.org, included a 9 code-execution backdoor inserted by a third party.

APRIL 2019

⊲ bootstrap-sass

An npm package contained code designed to steal cryptocurrency wallet seeds and other login

JUNE 2019 ⊲ 23 RubyGems packages Including chrome_taker, color_hacker, aloha_anal-

Someone removed a version of the library, bootstrap-sass v3.2.0.2 and immediately released a new version, moments later (v3.2.0.3) with malicious 10 code injected into it.

instruction details specific to cryptocurrency apps. Tipped off by npm researchers, makers of the Agama cryptocurrency wallets shifted $13 million worth of currency before adversaries could steal it.

JULY 2019 ⊲ libpeshnx

⊲ libpeshnx A PyPI package discovered to include a backdoor

A PyPI package discovered to include a backdoor vulnerability. While the package had been reported as containing a known vulnerability, it had not been removed from the Python package repository.

⊲ 230 RubyGems Pulled for typosquatting or impersonating popular

Pulled for typosquatting or impersonating popular open source packages.

AUGUST 2019 ⊲ 109 RubyGems

⊲ 109 RubyGems Yanked from the repository for typosquatting.

12 Yanked from the repository for typosquatting.

Adversaries compromised the account of a rest-client maintainer to install crypto miners in versions 1.6.10 to 1.6.13. Affected versions were downloaded about 1000 times. Similar vulnerabilities were found 13 in Gem packages: coming-soon and cron_parser.

NOVEMBER 2019 ⊲ sj-tw-test-security

⊲ lodahs, web3b, and web3-eht Taking advantage of a typosquatting exploit for

⊲ basic_authable Three versions of this Gems package released in

their malicious nature. ⊲ Octopus Scanner cookie/password stealing. 11

⊲ electron-native-notify (version 1.1.6)

Removed from the npm repository after it was discovered that it stole login information from the computers it was installed on and sent sensitive 14 information to a remote server.

All versions of the component contain malicious backdoor code that downloads and runs a script that

Taking advantage of a typosquatting exploit for lodash npm packages, all versions of the “lodahs” package contained malware designed to find and exfiltrate cryptocurrency wallets. web3b and web3- eht were removed for the same exploit pattern.

DECEMBER 2019 ⊲ python3-dateutil and jeIlyfish

⊲ python3-dateutil and jeIlyfish Two trojanized PyPI packages were caught stealing

Two trojanized PyPI packages were caught stealing SSH and GPG keys from the projects of infected developers. The two libraries imitated the popular 16 “dateutil” and “jeIlyfish” (the first L is an I).

The malicious npm package exfiltrates sensitive information such as hard-coded passwords or API access tokens through install scripts and targeting UNIX systems only.

⊲ 1337qq-js The malicious npm package exfiltrates sensitive

⊲ rest-client, coming-soon, and cron_parser

FEBRUARY 2020 ⊲ 381 RubyGems

Packages were yanked from the public repository 17 as a result of typosquatting concerns.

⊲ 381 RubyGems Packages were yanked from the public repository

26 open source packages were found to be compromised through malicious code injection. The malware was designed to enumerate and backdoor projects through the NetBeans IDE.

APRIL 2020

362 RubyGems

⊲ 362 RubyGems 14

⊲ 362 RubyGems

Software Supply Chain Attacks, July 2017 to July 2020 FIGURE 1D

Software Supply Chain Attacks, July 2017 to July 2020

J U 2017

npm credentials published online. Affects access to 14% of the npm repo (79K packages).

Malicious npm packaged typosquated. 40 packages harvested over two weeks, collecting credentials used to publish to the npm repository itself.

docker123321 images created on Docker Hub. Later accused of poisoning a Kubernetes honeypot (01/18) and equated to a crypto-mining botnet (05/18).

SEP 2017

PyPI typosquat: 10 malicious Python Evidence of the fake packages being credentials used to publish to the npm repository itself. (01/18) and equated to a crypto-mining botnet (05/18).

Linux distro hacked on GitHub. Unknown individuals gain control of the

J A N 2018

“I’m harvesting 10 malicious Python credit card numbers packages found. packages being credentials used to publish to the npm repository itself. David Gilbertson incorporated into writes a fictional tale (01/18) and equated to a crypto-mining botnet (05/18).

changelog ecosystem is and calling into question packages & sources. module, published in March.

Compromised on GitHub. JavaScript package repository gain control of the A hacker gains access Github Gentoo to a developer’s npm

F E B 2018

MAR 2018

Deleted go-bindata account credit card numbers resurrected by an unknown user. intentionally compromised. packages found. After a developer deleted their GitHub and passwords from your site. account, someone immediately grabbed the incorporated into writes a fictional tale software was noted multiple times

writes a fictional tale software was noted multiple times Back-doored PyPI package discovered. on his blog about Python module ssh-decorator back-doored between June and creating a malicious to enable theft of private ssh keys.

Adversaries compromised the account of a rest-client Affected versions (1.6.10 to 1.6.13) were downloaded about 1000 bb-builder removed from the

JUN 2018

Linux distro hacked Compromised

with a back-door

npm credentials resurrected by an unknown user. intentionally compromised. A malicious version of a

M A 2018

Back-doored npm package resurrected by an unknown user. intentionally compromised. discovered. npm security team responds account, someone immediately grabbed the package from a core to reports of a malicious back ID — inheriting the karma instilled in that ID contributor to the conventional-

resurrected by an unknown user. intentionally compromised. A malicious version of a from your site. account, someone immediately grabbed the package from a core Here’s how.” creating a malicious between June and to enable theft of private ssh keys. Sept 2017. npm package.

Back-doored Gems JavaScript package repository bootstrap-sass RCE caught stealing npm compromised. A malicious version of credentials. under 30 minutes the popular to a developer’s npm organization and through an account and injects modified the content exposed GitHub malicious code into a of repositories as well

N O 2018

OCT 2019

sj-tw-test-security All versions of the component maintainer to install Gems package “contain malicious back-door code released in 2017 crypto miners. were yanked from were downloaded about 1000 times. Similar vulnerabilities were found in Gem packages:

NOV 2019

basic_authable

Python3-dateutil 1337qq-js

Cryptocurrency attack via malicious code injection. bootstrap-sass RCE Malicious code targets users of Agama cryptocurrency wallets package discovered. package discovered. focusing on stealing the wallet seeds and login passphrases. passgen

passgen of the popular A RubyGems package discovered that contains a backdoor in its bootstrap-sass package, bootstrap-sass package, latest release that was used for cookie stealing. downloaded a total of downloaded a total of

AUG 2019

passgen of the popular A RubyGems package discovered that contains a backdoor in its

downloaded a total of downloaded a total of 23 RubyGems packages pulled from the public repository. 28 million times to date, 28 million times to date, popular JavaScript as pages within. All Packages were pulled from the public repository because they and with 1.6K library called eslint-code considered contained code for cryptomining or cookie/password stealing. dependencies, is scope, a sub-module of compromised.

DEC 2019

1337qq-js and jeIlyfish The malicious npm “contain malicious back-door code package exfiltrates Python libraries that downloads and runs a script that were caught the Gems repository stealing SSH and due to their malicious found in Gem packages: coming-soon and cron_parser. exploit for lodash npm packages, all versions of the “lodahs” package

JAN 2020

RubyGems 400 gems were packages yanked removed from the sensitive information from the public opens a reverse shell in the system such as hard-coded GPG keys from the projects or infected

Octopus Scanner 26 open source removed from the packages were found from the public public repository for opens a reverse shell in the system such as hard-coded repository as a allowing a remote attacker to passwords or API access tokens through projects or infected developers.

install scripts and concerns.

enumerate and back lodahs, web3b, and web3-eht

targets UNIX systems door NetBeans projects Taking advantage of a typosquatting

CHAPTER 1: OPEN SEASON ON OPEN SOURCE

Speed Remains Critical When

Responding to Legacy Software

Supply Chain Attacks

While bad actors are increasingly shifting their attention upstream, it is critical to understand and manage the software supply chain threats that remain prominent downstream. Specifically, organizations must establish a “rapid upgrade posture” so they can respond quickly to new zero-day disclosures by finding and fixing vulnerable open source dependencies in production applications.

Perhaps the best example of why this hygiene is so critical is the Equifax breach that began in March 2017. Following public disclosure from the Apache Foundation pertaining to a severe vulnerability in the popular Struts2 Framework, adversaries sprang into action and began exploiting the newly-known defect within 72 hours, well before many commercial IT teams (including Equifax) could respond and update their frameworks. This remarkably small window to respond led to numerous high-profile breaches, including Canada Statistics, Canada Revenue, the GMO Payment Gateway, Okinawa Power, Japan Post, India Post, and India’s AADHAAR digital identification system.

A similar exploit timeline played out with SaltStack this year. Vulnerabilities discovered in the open source application were announced on April 29th — along with safer, fixed versions. Within three days, 26 organizations that had not updated SaltStack lost control of their application to 19 adversaries (FIGURE 1F).

chapter will shed light on the industry’s growing understand. Our 2020 survey of 679 develop-

FIGURE 1E

Time to Remediate Known OSS Vulnerabilities After Detection Time to Remediate Known OSS Vulnerabilities After Detection

organizations become aware of new open source vulnerabilities within a day of public disclosure. Thirty five percent (35%) find out within one to seven days, and the remaining 48% become aware of new vulnerabilities after a week’s time.

Once an organization becomes aware of a new open source vulnerability, mitigating actions can begin. The same survey revealed that 51% of participants required more than a week to respond (FIGURE 1E). This means that adversaries averaging three days to exploit newly disclosed vulnerabilities hold an advantage over half their enterprise targets.

FIGURE 1F

Adversaries exploited open source vulnerabilities within 3 days of disclosure. Adversaries exploited open source vulnerabilities within 3 days of disclosure. 26 organizations breached in May 2020. 26 organizations breached in May 2020.

APRIL 29 MARCH 12 APRIL 15 SaltStack publishes version Vulnerability found in F- secure informs SaltStack of MAY 12 3000.2 and 2019.2.4 to fix SaltStack open source 6,000 publicly exposed Salt MAY 7 configuration MAY 2 Masters at risk of compromise. Cisco discovered the Censys reports the number issue and shares identifiers: framework, available LineageOS, a maker of an compromise of six of their stands at 2,928 Salt servers CVE- 2020- 11651 and CVE- as a PyPI package. open source operating Salt master servers, which still exposed — a 21% 2020- 11652. reduction from last week, system based on Android, are part of the Cisco VIRL- PE F- Secure: “We expect that According to Flexera, said it detected the intrusion (Internet Routing Lab and a 50% reduction Salt is used by around 17 on May 2nd at around 8 pm Personal Edition) service overall since the CVE was any competent hacker will be percent of organizations Pacific Time. infrastructure. announced. able to create 100% reliable exploits for these issues in with cloud deployments. under 24 hours.”

Update Before Exploits Begin

APRIL 30 MAY 2 MAY 3 MARCH 24 Sonatype ingests the CVE 15 breaches noted on GitHub accounts DigiCert SaltStack confirms receipt of information. • xiaopanggege : an unknown program suddenly ran today vulnerability report.

• atuchak : I have the same

• nepetadosmil : gents, this is an attack. We’ve had all firewalls disabled

• aidanstevens29 APRIL 23 SaltStack publishes advance notice to their users urging them not to expose Salt Masters to the internet and prepare to apply patch on April 29th.

reported that one of its Certificate Transparency logs was affected after attackers used the Salt exploits.

: a backdoor was also installed via the Ghost, a node.js blogging platform, exploit reports an attacker used a CVE in our

• ndmgrphc : entire system is being taken down SaltStack master to gain access to our

• nebev : been affected :( infrastructure and install a

• venugopalnaidu : we got the same issue cryptocurrency miner.

• gorgeousJ : same thing in my servers

• atastycookie : we are investigating Xen- Orchestra reports coin mining

• leeyo : we have the same problem script ran on some of their VMs tied to

• avasz : It also stopped and disabled docker services SaltStack vulnerability.

• aldenar : looking through my affected machines, a dropper Algolia scriptfile was found reports hackers installed a

• foobartenderbackdoor and a cryptocurrency miner : it also adds a key to /root/.ssh/authorized_keys on a small number of its servers.

• bruxy : same issue here

• mcpcholkin : I found it only on one server

• wavded : we had one job that was executed that did the following on each server

• justinimn : I got hit a few hours ago

• curu : Firewall rules stopped and disabled

• jblac : it's the same issue I was plagued with

• heruan : minor jobs are still spawning on minions 2020 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

CHAPTER 2 Open Source:

Open Source:

Supply and Demand

CHAPTER 2: OPEN SOURCE: SUPPLY AND DEMAND

JavaScript Package Downloads, JavaScript FIGURE 2A

JavaScript

One trillion JavaScript packages will be downloaded in 2020 based on monthly download volumes today. With over 86 billion package downloads in May 2020, the average monthly download traffic for npm packages has grown more 20 than 100% year over year. For the 10.7 million JavaScript developers around the world, this means each will download an average of 93,457 21 packages in ²⁰20. To keep pace with demand for component-based development, JavaScript community members introduced over 500,000 new component releases in the past year. There are now 1.3 million npm packages available to developers — up 63% from last year.

20B 17.5B Rolling Weekly Average 2013 – 2020 15B 12.5B BILLIONS

BILLIONS BILLIONS

2019 2019 2018 2018

2017 2017 Number of Download Requests for Number of Download Requests for Java Component Releases 2012 – 2020 2016 Java Component Releases 2012 – 2020 2016FIGURE 2B

2015

Number of Download Requests for Java Component Releases 2012 – 2020

2015

2014 2014

2013 2013

2012 2012

Java

There are an estimated 7.6 million Java developers worldwide. 22 In 2019, those developers triggered 226 billion open source software component download requests from Maven Central. Download request traffic was up 55% year over year, with the average developer requesting 29,736 component releases annually. With over 31 billion download requests in June, annual download requests for 2020 are on pace to top 376 billion.

For Java developers, the supply of Maven pack- ages increased from 3.7 million (June 2019) to over 5 million (June 2020). There are 337,000 Java open source projects that make their component releases available on Maven Central.

.NET DockerHub

.NET developers were also eager to consume According to stats available from the Docker Index, open source software packages over the past year. pulls of container images topped 8 billion for the Developers who downloaded an annualized 16.2 month of January. 25 This means annualized image billion NuGet packages in 2019 increased their pulls from the repository should top 96 billion appetite 177% to reach 44.8 billion annualized this year. 26 To keep pace with demand, suppliers downloads in 2020. 23 The supply of components pushed 2.2 million new images to DockerHub over increased by 700,000 package releases in the past the past year — up 55% since our last report. year — now totaling 2.3 million. 24 Over 200,000 open source projects now make their packages Now that we have examined supply and demand available on the NuGet Gallery. levels, our next chapter aims to shed light on attributes to look for when selecting the best open source projects to rely upon. ■

2020 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

CHAPTER 3 Identifying Exemplary

Open Source Suppliers

CHAPTER 3: IDENTIFYING EXEMPLARY OPEN SOURCE SUPPLIERS

Researching the Best

Performing OSS Projects

To better understand the health and habits of the open source component ecosystem, we researched thousands of Java components housed in The Central Repository (“Maven Central”) to help answer the following questions:

⊲ Do differences exist in how effectively OSS projects update their dependencies and fix vulnerabilities? Are there exemplary components that do this better than others?

⊲ Are exemplary components more widely-used than “non-exemplary” components?

608x faster at updating than “non-exemplary” components?

their dependencies clusters representing 8,201 open source projects

⊲ What factors correlate with exemplary components?

Components included in the research had to meet the following criteria:

⊲ Published to the Central Repository

⊲ Released at least two versions

⊲ Followed the Maven standard for versioning (e.g., correct use of numeric version strings, components separated by dots)

With a final data set of 24,053 components, we examined a number of attributes to identify relative hygiene across open source projects including, responsiveness to reported security vulnerabil- ities, number of dependencies, number of stale dependencies, frequency of releases, popularity, number of commits per month, developer team size, presence of continuous integration, and support type (foundation, commercial, or other).

SMALL EXEMPLARS

⊲ Has dependencies satisfying all of the above

FIGURE 3A

⊲ Has updated a dependency at least once

Finding Different

Behavioral Groups

As a result of our analysis, we identified five clusters representing 8,201 open source projects (FIGURE 3A).

Exemplars

We defined Exemplars to be those teams in the fastest 20% by Median Time to Update (MTTU) dependencies, and in the best (lowest) 20% by stale dependency count. Exemplars demonstrate statistically significant differences as compared to the rest of the data set in the following attributes:

⊲ 2.8x more commits

⊲ 1.5x more frequent releases

⊲ 2.9x fewer dependencies

⊲ 1.4x larger development teams

Large exemplars are 608x faster at updating their dependencies and they release 2.9x more frequently than non-exemplar clusters.

The smallest 50% of exemplary teams by number of developers have an average of less than two developers, but still manage to run popular, widely

at least a monthly basis), commit code frequently, release frequently, and do an excellent job of managing their dependencies. For example, we can see that large exemplary teams are 608x faster at updating their dependencies and they release 2.9x more frequently than non-exemplar clusters. We can see that 21% of these projects are associated with an open source foundation — a higher representation than any other cluster group.

CHAPTER 3: IDENTIFYING EXEMPLARY OPEN SOURCE SUPPLIERS

used, and high quality projects. However small in team size, they still update dependencies 475x faster than the rest of the population and are 4.3x more popular by download count compared to the Laggards and Cautious teams. Small projects were also 7x more likely to be commercially supported versus open source foundation supported.

Laggards

The teams in the bottom 20% in MTTU and stale dependencies are the furthest behind in terms of update hygiene. These teams release infrequently (around twice each year) and take on average almost two years to adopt updates to depen- dencies. The average period at least one of their dependencies is out of date is 203 days. They are 1.7x less popular (not downloaded as often as other projects on average). However, there are 288 projects in this group that are among the top 10% most downloaded projects from The Central Repository. This group represented 37% of our dataset.

FEATURES FIRST LAGGARDS These teams release frequently (top 50%) but

Cautious teams release new versions about

every two months, which is 1.3x more frequently

These teams release frequently (top 50%) but otherwise fall into the Laggard category (bottom 20% MTTU and stale dependencies). They have larger than average (2.4x larger) development teams than other Laggards, but do not prioritize upgrading dependencies. They release a new version every 29 days on average, but take an average of 501 days to upgrade dependencies when new versions are released. As a result, 88% of dependencies are out of date at release time. This was a small group, with 7% of the five cluster population exhibiting this behavior.

1.5x more frequent releases

Differentiate Through Exemplary OSS Projects Seven Performance Differentiate Through Metrics Seven Performance Metrics

250% more popular by download count

These teams maintain better-than-median update cadence, yet do not immediately adopt new versions of dependencies, choosing instead to wait a few months before moving to a new dependency release. This group represented 45% of our dataset falling into this category.

However small in team size, Small Exemplars still update dependencies 475x faster than the rest of the population.

Time to Remediate (TTR) vs. Time to Update (TTU)

FIGURE 3C

(cumulative percentage)

Time to Remediate (TTR) vs. Time to Update (TTU)

SOURCE: 2019 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

(cumulative percentage) 100% TTU Cumulative

90%

TTU mean: TTR Cumulative 80% 199 days

70% TTU median: 130 days 60% TTR mean: 50%326 days

40% Percentage of Population TTR median: **30%**180 days

20%

10%

FASTER SLOWER

Days to Update

to date on security updates, because security updates

Projects with Updated Teams should aim for a

are a subset of general updates. We observed that

Dependencies Are More Secure

many teams follow this practice, exhibiting very similarminimum of four releases The adoption curve for upgrading dependencies median times to remediate (MTTR) and mean time to and remediating vulnerabilities are similar, as update (MTTU) values. Large and small exemplars willannually and aim to shown in FIGURE 3C. When comparing MTTR generally achieve better security outcomes because

upgrade at least 80%

with MTTU for non-security-relevant updates of their strong MTTU performance (SEE FIGURE 3C). on a per-component basis, we see a correlation

of their dependencies

between update behavior for security relevant To adopt this practice, security managers should updates (MTTR) and non-security-relevant updates. encourage component and dependency updating with every release. practices by partnering with their development As we discovered in our 2019 report, developers stay-counterparts. ing up to date on dependencies will generally stay up

2020 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

CHAPTER 3: IDENTIFYING EXEMPLARY OPEN SOURCE SUPPLIERS

Guidance for Enterprise

Guidance for Open Source

Project Owners and Contributors

Given its association with good security practices and outcomes, we recommend a focus on accelerating and maintaining rapid MTTU. In addition to investing development effort on new features, bug fixes, etc., projects should commit similar resources to dependency management. This means that developers maintaining OSS projects who are considering adding a new dependency, and looking for a metric to guide that choice, would be wise to select dependencies with fast MTTU because such components naturally exhibit better security hygiene.

To progress comfortably into the status of Exemplar (top 80% of Exemplars), teams should aim for a minimum of four releases annually and aim to upgrade at least 80% of their dependen- cies with every release. A higher frequency of dependency updates statistically results in higher quality and more secure code.

managing 3,552 different projects and 11,294 unique releases (see Chapter 5) can introduce significant drag on development and is contrary to an enterprise’s need to develop faster as part of any agile, continuous delivery or DevOps practice.

Development Teams Enterprise development teams working with software supply chains often rely on an unchecked variety of supply from OSS projects where each developer or development team can make their own sourcing and procurement decisions. The effort of

Choosing open source projects should be considered an important strategic decision for enterprise software development organizations. Different components demonstrate healthy or poor performance that impacts the overall quality of their releases. Therefore, MTTU should be an important metric when deciding which components to utilize within your software supply chains. Rapid MTTU is associated with lower security risk and is accessible from public sources.

formalized procurement from public sources.

Just as traditional manufacturing supply chains intentionally select parts from approved suppliers and rely upon formalized procurement practices — enterprise development teams should adopt similar criteria for their selection of OSS compo- nents. This practice ensures the highest quality parts are selected from the best and fewest suppliers — a practice Deming recommended for decades. Implementing selection criteria and update practices will not only improve code quality, but can accelerate mean time to repair when suppliers discover new defects or vulnerabilities. Chapter 4 will further explore the impact of OSS component selection on overall application quality. ■

Just as traditional manufacturing supply chains intentionally select parts from approved suppliers and rely upon formalized procurement practices — enterprise development teams should adopt similar criteria for their selection of OSS components.

CHAPTER 4 How High Performance

How High Performance Teams Manage Open Source Software Supply Chains

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

We believe the results we found can help organi-

zations evaluate their approaches to using open

source components and improve the performance

and security of their software delivery practices.

companies with more than 5,000 developers. 63%

Survey of Open Source

of respondents were individual contributors or team and security of open Management Practices leads, while 37% were managers, VPs, or executives. Participants achieved a 75% completion rate,

Analyzing the performance and security of open source component-based software development is made easier because, similar to manufacturing Industry Verticals Analyzed supply chains, the inventory is visible. for OSS Component-Based

Development Practices For this year’s report, we expanded our survey of OSS component-based development practices to include 679 engineering professionals employed in commercial roles. We inquired about software delivery outcomes (e.g., deployment frequency, security, engineering productivity, job satisfaction) and practices (e.g., approaches and philosophies to utilizing open source components, organizational design, governance, approval processes, and tooling). The goal was to discover to what extent various practices contribute to success. To assess this, we performed a number of analyses including fitting regression models to the data, clustering, and examining statistically-significant betweengroup effects.

We created a survey with 41 questions, exploring ten areas of software outcomes (dependent variables), and twenty-four areas of software practices, tooling, organization, policies, etc. (independent variables).

FIGURE 4A Industry Verticals Analyzed

To identify cohorts with similar reported outcomes, and identify high and low performers, we used 27 a cluster analysis. We found four clusters with markedly different levels of performance, with different patterns of practices, and with almost all factors being statistically different. We labeled them as follows:

FIGURE 4A) . Organizations of all sizes were repre-

Analyzing the performance and security of open source component-based software development is made easier because, similar to manufacturing supply chains, the inventory is visible.

Cluster Analysis and Findings

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

⊲ Security First: low productivity, great risk management outcomes (N=167)

We can quickly see the different characteristics of the four clusters by projecting them onto a quadrant —  on one axis are all the productivity-related

⊲ Productivity First: high productivity, poor risk management outcomes (N=103)

outcomes combined into a single dimension, and

productivity. The Security First cluster has high security outcomes, but low productivity, and the Productivity First has high productivity, but poor security outcomes.

combined into a single dimension (both using principal components analysis).

The resulting graph (FIGURE 4B) identifies a High Performers cluster (purple, upper right) who demonstrate superior risk management outcomes while maintaining high levels of productivity. The Low Performers cluster (red, lower left) identifies the opposite pattern: demonstrating substandard risk management outcomes and low levels of

It is important to note that the High Performers achieved even higher average productivity levels than the Productivity First cluster. As seen in FIGURE 4B, the High Performers are tightly clustered in the upper right quadrant, while the Productivity First group is more distributed across the bottom left- and right-quadrants.

FIGURE 4B

Measuring Risk Management vs. Productivity Outcomes Measuring Risk Management vs. Productivity Outcomes

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Comparing High Performers vs. Low Performers

The tables on the following pages show how decisively the High Performers outperform the low performers in software delivery and security —  they deploy more frequently, they detect and remediate vulnerable OSS components more quickly, onboard developers onto new teams more quickly, and approve new OSS components for use more quickly.

Furthermore, High Performers are more confident about the compliance and security of their OSS components, and have fewer problems updating their OSS components. Compared to Low Performers, High Performers are.

FIGURE 4C

Comparing High Performers vs. Security First

Many have argued that effective risk management practices are always at the expense of developer productivity, (i.e., “better security slows down development”). We can see these outcomes in

⊲ 4.9x less likely to have dependencies break application functionality

⊲ 3.8x more likely to describe updating dependencies as easy (i.e., not painful) ⊲ 33x more likely to be confident that OSS depen-

15x more frequent deployments

Performers against

⊲ 33x more likely to be confident that OSS dependencies are secure (i.e., no known vulnerabilities)

26x faster DETECTION of vulnerable OSS components

⊲ 2.1x more likely to have access to newer OSS component versions where prior defects have been fixed

26x faster REMEDIATION of vulnerable OSS components

5.7x

5.7x less time required for developers to be productive when SWITCHING teams

26x

⊲ 77% more likely to automate approval, management, and analysis of dependencies

less time to APPROVE a new OSS dependency for use

To better understand these differences, we examined what practices separate the High Performers from this Security-First cluster. It turns out that High Performers tend to have a governance structure that relies much more heavily on automated tooling. Compared to the Security-First group, the High Performers were:

⊲ 59% more likely to be using software composition analysis (SCA) tools

seemed to be achieving good security outcomes

outcomes can be achieved simultaneously.

⊲ 96% more likely to be able to centrally scan all deployed artifacts for security and license compliance.

⊲ 56% more likely to have centrally-managed CI infrastructure

their organizations as a great place to work

of a Software Bill of Materials (SBOM) for each

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Variables Most Impacting Performance and Risk Management

In this section, we state all of the hypotheses we had when we designed the survey, and state which practices (independent variables) we believed would affect the performance outcomes (dependent variables) — we also define how we measured them.

To better understand the connection between practices and outcomes, and potentially understand how one can improve performance, we fit a 28 linear model to the data. We measured and, where appro- 2 priate, report r values, which describe the proportion of variance in each outcome explained by the model and describe the top practices, based on their contribution to increases in the ²9 30 outcome being analyzed.

Mean Time to Remediate (MTTR)

(All independent and dependent variables are listed and described in Appendix B.)

Influencing Risk Management Outcomes

Across all the risk management outcomes, the most consistent factors associated with positive risk management outcomes were:

⊲ Having a clear process for adding and removing OSS dependencies

⊲ Remediating known OSS vulnerabilities as a regular part of development

⊲ Updating OSS dependencies regularly

⊲ Using SCA tooling and incorporating this tooling into CI

INFLUENCING RISK MANAGEMENT OUTCOMES: Mean Time to Detect Vulnerabilities (MTTD)

CHAPTER 4. HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

INFLUENCING RISK MANAGEMENT OUTCOMES: OSS Security Measured as the level of confidence that applications are not using open source components with known vulnerabilities.

Measured as the level of confidence that applications are not using open source components with known vulnerabilities.

INFLUENCING RISK MANAGEMENT OUTCOMES: License Compliance Measured as the level of confidence that the development team is in compliance with the organization’s policies regarding open source licenses.

Measured as the level of confidence that the development team is in compliance with the organization’s policies regarding open source licenses.

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Factors which influence software delivery productivity are notoriously elusive, although the State of DevOps Report has wonderfully illuminated its link to continuous delivery, culture, lean product development, etc. Our intent was to further explore other practices that could improve aspects of productivity, which revealed some surprises.

INFLUENCING PRODUCTIVITY OUTCOMES: Developer Portability

Measured as the time required for developers to reach normal productivity when switching teams.

OSS Component Approval Times

Measured as the time it takes for developers to get a new OSS library approved for use.

INFLUENCING PRODUCTIVITY OUTCOMES: Internal Forks Measured as how common it is for internally modified versions of open source projects to be maintained.

Measured as how common it is for internally modified versions of open source projects to be maintained.

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Guidance for Enterprise

High Performer results are achieved not by

Development Teams

implementing a single tool or practice, but

through a combination of culture, development

Our research shows that faster innovation and

practices, policy enforcement, automation, and

better risk management are not mutually exclu-

integrations applied across the development

are accelerating velocity while simultaneously

Influencing Job Satisfaction

This year’s survey measured job satisfaction by five questions about various aspects of work including organizational support, level of fit between skills and tasks, and ability to complete work.

| INFLUENCING JOB SATISFACTION: Work Attitudes and Motivation

Interestingly, the most predictive question of job satisfaction was “How is your current open source risk management initiative resourced and supported?” This was the most detailed question we asked regarding general organizational support and included sub-questions about executive support, budget, tooling, and documentation. We suspect that this relationship is highlighting a connection between level of employee support and job satisfaction rather than an effect specific to support of open source risk management initiatives. In future surveys we will ask more general “organizational support” questions to evaluate this hypothesis.

Security First teams desiring to transform

Our investigation into measures of high performance component-based software development and delivery helped us confirm four overarching, compelling and predictable criteria: time to update depen- dencies, deployment frequency, time required for developers to be productive when switching teams, and time to detect and remediate defective components. Teams striving for productivity and risk management outcomes that improve management of their software supply chains and delivery practices should track performance of these criteria.

themselves into High Performers would benefit from automating their approval, management and analysis of open source components. They should also consider integrating developer friendly SCA tools into their CI process so they can automatically scan build artifacts, easily identify open source security and licensing risk, and benefit from a SBOM for all applications.

job satisfaction are very similar to the early results

Forsgren, Jez Humble, Gene Kim, 2015 Puppet

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Productivity First teams wanting to shift up into

Patterns Across OSS

the High Performer quadrant should prioritize

Component Updates: Easy,

partnering with governance counterparts to Difficult, and Planned integrate automated security scanning into their CI process so they can easily add and remove OSS

FIGURE 4D

Stronger Risk Management and Productivity Outcomes for High Performers (Comparison of Cluster Centroids)

Over the years, we’ve become increasingly convinced that while updating dependencies is very important for functionality and security, there is a huge economic cost to staying up-to-date. Ideally, dependencies should be updated, simply, safely and painlessly, and as part of the routine development process. But reality shows that this ideal is rarely met.

An astonishing story of how far an organization can stray from ideal update practices comes from Eileen M. Uchitelle, Staff Engineer at GitHub, who described how it took seven years to successfully migrate 32 GitHub from a forked version of Rails 2 to Rails 5. Even with new tools available to developers that automatically create pull requests with updated dependencies, changes in APIs and potential breakage can still hold back many developers from updating. We suspect this change-induced breakage is a primary driver of poor updating practices.

CHAPTER 4: HOW HIGH PERFORMANCE TEAMS MANAGE OPEN SOURCE SOFTWARE SUPPLY CHAIN

Taking a deeper dive into the vast data available to us from The Central Repository, we can better visualize open source project releases and their adoption by enterprise application development teams who migrate from one version to a newer one. We believe this data shows how OSS component selection can play a major role in allowing for easier and more frequent updates.

The following graphs show the different stories around OSS update patterns by software development teams. Updates from one version of a library to another are visually depicted by connecting the two versions with an arc. The horizontal axis is an ordered list of library releases, where version numbers increase as you move right.

sufficiently difficult that the effort must be planned

and some version ranges end up being avoided.

Consider the graph for the widely used joda-time library (FIGURE 4E), which shows that developers using this OSS component update fairly uniformly between all pairs of versions. This suggests that updates are easy, presenting a seemingly homogenous set of versions to select migrate to and from.

On the opposite extreme, consider the graph for the hibernate-validator library (FIGURE 4F), where there are two sets of communities using it —  one favoring version 5 and another preferring version 6. The two communities very rarely intersect. This suggests that updating to version 6 from version 5 is either too difficult, or the value is not worth the effort.

In our future work, we would like to further investigate which dependencies the High Performers and other notable clusters are using and the criteria they use to select them, while measuring the effort and cost required to stay up-to-date. We believe that this could reveal lessons and principles that could help every organization using open source software components.

Now that we have explored practices and related outcomes that contribute to successful software supply chain management, let’s take a closer look at the volume, quality, and security of open source component consumption in the enterprise. ■

Finally, we take a look at the pattern for spring-

FIGURE 4E joda-time.joda-time library

MIGRATION PATTERNS BETWEEN OSS COMPONENT RELEASES

hibernate-validator library

Poor Migrations -> Good Migrations (by application count)

50+ 40 30 20 10 0 10 20 30 40 50+

FIGURE 4F

hibernate-validator library

Poor Migrations -> Good Migrations (by application count) 50+ 40 30 20 10 0 10 20 30 40 50+

spring.spring-core libraryMIGRATION PATTERNS BETWEEN OSS COMPONENT RELEASES

Poor Migrations -> Good Migrations (by application count)

FIGURE 4G 50+ 40 30 20 10 0 10 20 30 40 50+

spring.spring-core library

Poor Migrations -> Good Migrations (by application count) 50+ 40 30 20 10 0 10 20 30 40 50+

MIGRATION PATTERNS BETWEEN OSS COMPONENT RELEASES

2020 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

CHAPTER 5 The Trust and

The Trust and Integrity of Software Supply Chains

CHAPTER 5: THE TRUST AND INTEGRITY OF SOFTWARE SUPPLY CHAINS

Enterprise development teams often rely on an unchecked variety of supply from OSS projects where each developer or development team can make their own sourcing and procurement decisions**. Development teams have an inherent** trust in their OSS component’s authenticity and integrity. Yet the complexity of multi-layered open source software supply chains can obfuscate risk for those seeking to avoid it.

Choosing open source projects should be considered an important strategic decision for enterprise software development organizations. Just as traditional manufacturing supply chains intentionally select parts from approved suppliers and rely upon formalized procurement practices — enterprise development teams should adopt similar criteria for their selection of OSS components to ensure the highest quality parts are selected from the best and fewest suppliers.

As Jim Zemlin, Executive Director of the Linux Foundation recently remarked, “Open source is an undeniable and critical part of today’s economy, providing the underpinnings for most of our global commerce. Hundreds of thousands of open source software packages are in production applications throughout the supply chain, so understanding what we need to be assessing for vulnerabilities is the first step for ensuring long-term security and 33 sustainability of open source software.”

1 in 10 OSS Downloads

Are Vulnerable

To better understand how defective and known vul-

FIGURE 5A Development’s Visibility, Awareness, and

Development’s Visibility, Awareness, and Development’s Visibility, Awareness and Control of its Software Supply Chain Control of its Software Supply Chain

Reduction of Visibility, Awareness, and Control

For the past seven years Sonatype has analyzed the patterns and practices associated with Java components being downloaded from The Central Repository (FIGURE 5B). In 2019, 10.4% of the billions of downloads had at least one known vulnerability.’

source repositories (e.g., Maven Central, npmjs. org, RubyGems.org, NuGet Gallery). Developers

FIGURE 5B

Furthermore, research from the University of Darmstadt published in August 2019 revealed that nearly 40% of all npm packages rely on code with known vulnerabilities. Perhaps even more concerning is that 66% of security vulnerabilities in npm packages remain unpatched, leaving developers who want to use secure packages with no safe 34 alternatives.

CHAPTER 5: THE TRUST AND INTEGRITY OF SOFTWARE SUPPLY CHAINS

OSS Components Make Up

FIGURE 5C Construct of a Modern Application Construct of a Modern Application

FIGURE 5D The OSS Knowledge Gap Widens Over Time The OSS Knowledge Gap Widens Over Time

Enterprises Rely on Code From 3,500 Suppliers, But Quality Varies

Further analysis of downloads from those organizations reveals that 30,862 (8.3%) included at least one known security vulnerability. Just as well, not all security vulnerabilities are created equal. Of the 30,862 vulnerable downloads, 68% had Common Vulnerability Scoring System (CVSS) at 7.0 or above on a 10 point scale. Thirty percent (30%) had CVSS scores above 9.0 on a 10 point scale. Minor fluctuations in the percentage of vulnerable downloads were seen on a country by country basis: United States (8.6%), France (8.3%), United Kingdom (8.6%), and Germany (7.81%).

Just because a developer downloaded a component does not mean that it was used in an application. To better understand how many open source components were used by developers, we investigated and analyzed 1,700 applications for this year’s report. We found that development teams use an average of 135 software components of which 90% are open source. It was not uncommon to see applications assembled from 2,000 – 4,000 OSS component releases.

code. Our study of 15,000 enterprise software

development organizations revealed an average

of 373,000 open source component downloads

annually. The downloads represent an average

of 3,552 OSS projects — the external supplier network for code serving modern enterprise

CHAPTER 5: THE TRUST AND INTEGRITY OF SOFTWARE SUPPLY CHAINS

While any developer knows that open source components are used to build an application, the enterprise does not carry the same awareness. An analysis of open source component use in organizations by Revenera is telling of software supply chain awareness. In 2018, development teams using open source in development disclosed their awareness of 29 OSS being used while audits of their environments revealed 626 35 components — a 22x difference! (FIGURE 5D)

components — a 22x difference! (FIGURE 5D) 35

21% of Enterprises Experienced

Open Source Breaches

According to the X-Force Threat Intelligence Index attacks on known vulnerabilities increased to 30% in 36 2019, up from 8% the previous year. Development teams relying on open source components that sometimes contain known vulnerabilities were not immune to these attacks. The 2020 DevSecOps Community Survey of over 5,000 development professionals revealed that 21% had experienced an open source component related breach in the past 12 months (FIGURE 5E). ■

Open source component related breaches FIGURE 5E continue to drop, but still occur much too often.

had at least one known security vulnerability.

On average, the applications contained 38 known

continue to drop, but still occur much too often. Open source component related breaches SOURCE: 2020 DEVSECOPS COMMUNITY SURVEY, SONATYPE continue to drop, but still occur much too often.

SOURCE: 2020 DevSecOps Community Survey, Sonatype

CHAPTER 6 The Changing OSS

The Changing OSS Landscape: Social Activism and Government Standards

CHAPTER 6: THE CHANGING OSS LANDSCAPE: SOCIAL ACTIVISM AND GOVERNMENT STANDARDS

Social Activism and Open

Source Software

Social activism has been high on the agenda of many in the tech community. Developers at Google, WeWork, Kickstarter, Amazon, and other companies across the tech industry have been more active at protesting employer decisions, petitioning them to abstain from doing business with government agencies, and denouncing unfair treatment of employees.

In September 2019, in an effort to protest his former employer’s commercial relationship with the U.S. Immigration and Customs Enforcement (ICE), Seth Vargo removed his “Sugar” code from GitHub and the RubyGems repository. The missing code was eventually replaced, but not before a significant portion of Chef’s customers were impacted without warning. Addressing the community as to why he pulled his code, Vargo wrote “I have a moral and ethical obligation to prevent my source [code] 37 from being used for evil.”

January 2020 surfaced another protest o f sorts when Nikolay Kim deleted his actix-net and actix-web open source project from their public repos after being harassed too many times by his user community. He declared “Being a maintainer of a large open source project is not a fun task... I am done with open source” and “I moved

automated builds relying on the code causing many in the community to panic. Kim’s code was eventually restored to public repos after he transferred ownership to another developer in the 38 community.

As discussed in Chapter 4, successful productivity outcomes for High Performers were tied, in part, to keeping a centralized record of applications, their dependencies, and the associated development teams. Given its association with good outcomes, we recommend the use of repository managers to proxy public OSS repositories and host OSS components locally. Locally hosting any components needed by developers will help improve business continuity during future protests or actions by activists.

from protests. In 2017, a developer harassed by

corporate lawyers pulled his left-pad code from

the npm repository temporarily “breaking the

internet” as numerous automated build environ-

Governments Apply New

Standards to Secure

Software Supply Chains

Secure software practices extend from early development through the active life of an application in the market. With an ever increasing number of application breaches occurring, standards bodies and governments are stepping in to hold development organizations accountable for the quality and security of the code they assemble and build.

In April 2019, the Open Chain Specification, version 2.0, was published to define the key requirements of a quality open source license compliance program. The objective was to provide a benchmark that builds trust between organizations exchanging software solutions composed of open source software.

github account.” His action immediately impacted

United States

OPEN CHAIN PROJECT —

LINUX FOUNDATION

— SETH VARGO

“ I have a moral and ethical obligation to prevent my source [code] from being used for evil.”

CHAPTER 6: THE CHANGING OSS LANDSCAPE: SOCIAL ACTIVISM AND GOVERNMENT STANDARDS

Section 3.1 of the specification called for creating a Software Bill of Materials (SBOM). The SBOM would be used to identify, track, review, approve, and archive information about the open source software components used in a software application, middleware, firmware or operating system. The specification maintains that an SBOM is needed to support the systematic review and approval of each component’s license terms to understand the obligations and restrictions as it 39 applies to the distribution of software.

CYBERSECURITY & INFRASTRUCTURE SECURITY AGENCY

In May 2019, CISA’s Supply Chain Risk Management (SCRM) published a guide for detailing actionable steps on how to start securing software supply chains. Steps recommended building a list of the software components organizations procured, mapping supply chains to better understand what components were being procured, determining how organizations would assess the security culture of suppliers, and establishing systems for checking supply chain practices against 40 guidelines.

U.S. CONGRESS

In July 2019, Sen. Mike Crapo (R-ID) and Sen. Mark Warner (D-VA) introduced a bill explaining that software supply chains have proven to be major means through which adversaries seek gain access to weapons systems, IT systems, and communications technology platforms. While not signed into law, the bill had called for “stronger effort should be placed on securing the vast supply chains of the contractors responsible for developing and producing the defense related capabilities of the 41 United States.”

OASIS OPEN COMMAND AND CONTROL (OPENC2) TC

In October 2019, members of the OASIS Open Command and Control (OpenC2) TC started sharing documents, specifications, lexicons or other artifacts on GitHub aimed to fulfill the needs of cyber security command and control in a standardized manner. Among them, the Department of Defense comply-to-connect use case defined an early step of querying the new device requesting its “Software Bill of Materials” and comparing it to 42 policy as part of an acceptance process.

NATIONAL TELECOMMUNICATIONS AND INFORMATION ADMINISTRATION

Over the past year, the National Telecommunications and Information Administration (NTIA) continued its pursuit to establish the definition formats and standards for a Software Bill of Materials (SBOM). This multiyear, non-partisan initiative aims to define SBOM concepts and related terms, offers a baseline of how software components are to be represented, and discusses the processes around SBOM creation. The initiative has also detailed the benefits of building and managing SBOMs from the perspective of those who make software, those who choose or buy software, and those who operate it — characterizing security, quality, efficiency, and other organizational benefits.

Working hand in hand with the U.S. Food and Drug Administration (FDA), the NTIA produced a report documenting the successful execution and lessons learned of a proof-of-concept exercise led by medical device manufacturers (MDMs) and healthcare delivery organizations (HDOs). The exercise examined the feasibility of SBOMs being generated by MDMs and used by HDOs as part of

United States.” 41

In July 2019, Sen. Mike Crapo (R-ID) and Sen. Mark Warner (D-VA) introduced a bill explaining that software supply chains have proven to be major means through which adversaries seek gain access to weapons systems, IT systems, and communications technology platforms.

CHAPTER 6: THE CHANGING OSS LANDSCAPE: SOCIAL ACTIVISM AND GOVERNMENT STANDARDS

operational and risk management approaches to 43 medical devices at their hospitals.

NATIONAL DEFENSE AUTHORIZATION ACT FOR FISCAL YEAR 2020 In December 2019, the NDAA — now signed into

In December 2019, the NDAA — now signed into law, called for the U.S. Secretary of Defense to establish pathways for the efficient and effective acquisition, development, integration, and timely delivery of secure software. The Act included the requirement for software security testing that includes vulnerability scanning and also asks for the establishment of DevSecOps practices inside the Department of Defense.

Section 800 of the Act required “assurances that cybersecurity metrics of the software to be acquired or developed, such as metrics relating to the density of vulnerabilities within the code of such software, the time from vulnerability identification to patch availability, the existence of common weaknesses within such code, and other cybersecurity metrics based on widely-recognized standards and industry best practices, are generated and made available to the Department of Defense and the congressional 44 defense committees.”

In April 2020, NIST released new standards for improving software security aimed at helping “software producers reduce the number of vulnerabilities in released software, mitigate the potential impact of the exploitation of undetected or unaddressed vulnerabilities, and address the root causes of vulnerabilities to prevent future 45 recurrences.”

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY In April 2020, NIST released new standards for

⊲ Create and maintain a software bill of materials (SBOM) for each OSS component used and every proprietary software package created.

⊲ Securely archive a copy of each release and all of its components (e.g., code, package files, OSS and third-party libraries, documentation), and release integrity verification information.

⊲ See if there are publicly known vulnerabilities in the OSS software components and services that the vendor has not yet fixed.

⊲ Ensure each software component is still actively maintained, which should include new vulnerabilities found in the software being remediated.

⊲ Determine a plan of action for each third party and OSS software component that is no longer being maintained or available in the future.

⊲ Use the results of commercial services for vetting OSS software components.

⊲ Establish an organization-wide software repository to host sanctioned and vetted OSS components.

party components are components.

⊲ Maintain a list of organization-approved commercial OSS components and component versions.

this is likely to also NATIONAL INSTITUTE OF

United Kingdom

United Kingdom or unaddressed vulnerabilities, and address the

CENTRE: SECURE DEVELOPMENT 45

THE NATIONAL CYBER SECURITY CENTRE: SECURE DEVELOPMENT AND DEPLOYMENT GUIDANCE The Centre recognized that software development

The Centre recognized that software development practices are becoming increasingly automated and reliant on open source and third party

New guidance released by the Centre advised that “third party coding frameworks and libraries also need to be considered in the same light as the code you author. If third party components are themselves vulnerable, this is likely to also impact your system.”

CHAPTER 6: THE CHANGING OSS LANDSCAPE: SOCIAL ACTIVISM AND GOVERNMENT STANDARDS

components. New guidance released by the Centre advised that “third party coding frameworks and libraries also need to be considered in the same light as the code you author. If third party compo- nents are themselves vulnerable, this is likely to 47 also impact your system.”

also impact your system.” 47

In an effort to help development teams evaluate their OSS components and reduce security risk, the Centre provided the following eight questions:

⊲ If there is a security vulnerability in the third party components of your code, what security impact may this have on your system?

⊲ Is the dependency actively developed and

⊲ If a vulnerability is found in one of your depen- dencies, would you know? Who would fix it?

⊲ Are you using any old versions of third party code known to contain security vulnerabilities?

⊲ If the third party dependency you are using is configurable, consider disabling or removing unneeded functionality which may widen the 48 attack surface of your product.

⊲ Do you know anything about the author and maintainer of the dependency? How do they view and approach security?

FIGURE 6A SOURCE: The Australian Cyber Security Centre (ACSC)

⊲ Does the dependency have any history of security vulnerabilities? What’s important here is not necessarily that issues are discovered, but how they are handled.

Australia

This year, the Australian Cyber Security Centre (ACSC) has developed prioritised mitigation strategies to help organizations mitigate cyber security incidents caused by various threats (SEE FIGURE 6A). The Centre defined mitigation strategies that could be applied along three maturity levels. For updating third party libraries and patching applications, the guidance recommended mitigating actions within a month at the lowest maturity level and

within 48 hours at the highest maturity level, while also recommending automated tooling to track where and when cybersecurity updates had been 49 performed. ■

your product during the build or deployment

process, can you ensure that it can’t be maliciously modified? You could achieve this by verifying its origin and integrity, for example.

Summary

We’ve observed double and triple digit growth in open source component ecosystems for over a decade. The industry eclipsed 10 billion open source component downloads in 2012 and within five years witnessed 100 billion download requests. With no slowdown in sight, 2020 is on pace to surpass 1.5 trillion download requests.

The purpose of our 6th annual report was to share evidence, practices and outcomes we observed across software supply chains — upstream and downstream. Our findings are clear. Productivity does not have to come at the cost of reduced security.

On the supply side, we observed that Exemplary open source projects benefit tremendously from more frequent code commits, dependency updates and releases. The more frequent the updates, the generally more secure the OSS project.

On the demand side, we discovered a range of enterprise practices that influenced successful software supply chain outcomes. High Performers deployed more frequently, detected and remedi- ated vulnerable OSS components more quickly, and approved new OSS components efficiently. The High Performers also onboarded developers onto new teams faster and their employees demonstrated high levels of satisfaction on the job.

Thank you for reading this year’s report. Please

its data, perspectives, and insight. We welcome any feedback that would help us improve our future

Our deep examination of consumption patterns, development practices, and cybersecurity hygiene revealed:

⊲ 929 next-generation cyber attacks actively targeting OSS projects over the past year (Chapter 1)

⊲ 608x faster median time to update dependencies and 2.9x more frequent releases for large exemplary OSS projects compared to non-exemplar clusters (Chapter 3)

⊲ 26x detection and remediation of open source vulnerabilities by high performance teams (Chapter 4)

⊲ 11% of OSS components used in applications had at least one known security vulnerability (Chapter 5)

⊲ 21% of development teams experienced an open source related breach in the past 12 months (Chapter 5)

It is encouraging to see exemplary OSS projects and innovative enterprise development teams are delivering high quality, security software at a rapid pace. Their dedication and results are not rare and their performance serves as a benchmark for others to strive for and achieve.

Sources

1 https://blog.sonatype.com/open- source-software-is-under-attack- new-event-stream-hack-is-latest- proof

2 https://blog.sonatype.com/ octopus-scanner-compromises-26- oss-projects-on-github **3 https://arxiv.org/pdf/2005.09535.** pdf 4 https://www.usenix.org/system/ files/sec19-zimmermann.pdf

5 https://www.usenix.org/system/ files/sec19-zimmermann.pdf

6 https://www.zdnet.com/article/ the-linux-foundation-identifies- the-most-important-open-source- software-components-and-their- problems 7 https://medium.com/@bertusk/ discord-token-stealer-discovered- in-pypi-repository-e65ed9c3de06 **8 https://github.com/dasfreak/** Backstabbers-Knife-Collection **9 https://www.cvedetails.com/cve/** CVE-2019-14282 10 http://dgb.github.io/2019/04/05/ bootstrap-sass-backdoor.html

11 https://github.com/rubygems/ rubygems.org/issues/2034

12 https://github.com/rubygems/ rubygems.org/wiki/Gems-yanked- and-accounts-locked#19-aug-2019 13 http://arstechnica.com/ information-technology/2019/08/ the-year-long-rash-of-supply-chain- attacks-against-open-source-is- getting-worse

14 https://www.npmjs.com/ advisories/1119

15 https://www.npmjs.com/ advisories/1308

16 https://www.zdnet.com/article/ two-malicious-python-libraries- removed-from-pypi 17 https://gist.github.com/colby-sw andale/11dadff435b02f887fc68178 cd4fb0dc **18 https://www.theregister.** com/2020/04/21/rubygems_ bitcoin_malware

19 https://blog.sonatype.com/ saltstack-20-breaches-within-four- days **20 https://www.npmjs.com/** 21 https://www.daxx.com/blog/ development-trends/number- software-developers-world 22 https://www.zdnet.com/article/ programming-languages-python- developers-now-outnumber-java- ones

**23 https://www.nuget.org** **24 https://www.nuget.org** 25 https://www.docker.com/blog/ introducing-the-docker-index

**26 https://devclass.com/2020/** 02/05/docker-knits-together-hub- stats-says-pulls-over-8-billion/

27 Exploratory clustering was initially done with the SPSS two- step clustering method, and later was performed with k-means using SciKit Learn, starting from random

cluster centers and taking the best of 50 runs. Highly correlated variables were first converted to single dimensions with principal components analysis. 28 We used the SciKit Learn (version 0.21.1) implementation of elastic net regression with alpha=0.1 and an L1 ratio of 0.7. 29 Higher r² is better and indicates that the model explains more of the change in outcome. 30 A caveat: our survey relies on self-reported data and we did not have access to direct measures of the behavior.

31 https://puppet.com/resources/ report/2015-state-devops-report

32 Eileen M. Uchitelle, “The **Past, Present, & Future of Rails at GitHub”: https://speakerdeck.** com/eileencodes/railsconf- and-balkan-ruby-2019-the-past- present-and-future-of-rails-at- github and “RailsConf 2019 — The Past, Present, and Future of Rails at GitHub,” 25 minutes in, https://www.youtube.com/ watch?v=vIScxVu00bs

33 https://www.zdnet.com/article/ the-linux-foundation-identifies- the-most-important-open-source- software-components-and-their- problems/ 34 https://www.usenix.org/system/ files/sec19-zimmermann.pdf

**35 https://info.flexerasoftware.com/** SCA-Ebook-Maturity-Open-Source- Software

36 https://securityintelligence.com/ 47 https://www.ncsc.gov.uk/ posts/x-force-threat-intelligence-collection/developers-collection/ index-reveals-top-cybersecurity-principles/produce-clean- risks-of-2020/ maintainable-code

37 https://github.com/sethvargo/ 48 https://www.ncsc.gov.uk/ chef-sugar collection/developers-collection/ principles/produce-clean- **38 https://devclass.** maintainable-code com/2020/01/20/rust-framework- dev-says-im-done-with-open-49 https://www.cyber.gov.au/acsc/ source-has-second-thoughts view-all-content/publications/ essential-eight-maturity-model **39 https://wiki.linuxfoundation.** org/_media/openchain/ openchainspec-2.0.pdf

40 https://www.cisa.gov/sites/ default/files/publications/ict_scrm_ essentials_508.pdf 41 https://www.congress.gov/116/ bills/s2316/BILLS-116s2316is.pdf

42 https://github.com/oasis-tcs/ openc2-usecases/blob/master/ Cybercom-Plugfest/uc-A-comply-to- connect.md

43 https://www.ntia.gov/ files/ntia/publications/ framingsbom_20191112.pdf 44 https://www.congress.gov/116/ bills/s1790/BILLS-116s1790enr.pdf

45 https://csrc.nist.gov/ publications/detail/white- paper/2020/04/23/mitigating-risk- of-software-vulnerabilities-with- ssdf/final **46 https://nvlpubs.nist.** gov/nistpubs/CSWP/NIST. CSWP.04232020.pdf

2020 STATE OF THE SOFTWARE SUPPLY CHAIN REPORT

Appendix A

Acknowledgments

Each year, producing the State of the Software Supply Chain report is labor of love. It is produced to shed light on the patterns and practices associated with open source software development. We began collecting data for our 2020 report from the moment our 2019 report was published.

report was published. The report is made possible thanks to a tremendous effort put forth by many team members at Sonatype, including: Derek Weeks, Matt Howard, Joel Orlina, Bruce Mayhew, Gazi Mahmud, Dariush Griffin, Brian Fox, AJ Brown, Ember DeBoer, Mike Donovan, Cameron Townsend, Ilkka Turunen, Alexis Del Duke, Elissa Walters, Adam Cazzolla, Keith Sprochi, and Neil Donewar.

We would also like to offer thanks for contributions big and small from: Hasan Yasar (Carnegie Mellon University Software Engineering Institute), DJ Schleen (Rally Health), and others across the DevOps and open

Finally, we could not have produced this report without the amazing contributions and countless hours of deep analysis from our research partners Gene Kim from IT Revolution and Dr. Stephen Magill, CEO of MuseDev.

A very special thanks goes out to Melissa Schmidt who created the incredible design for this year’s report.

Appendix B

About the Analysis

The authors have taken great care to present statistically significant sample sizes with regard to component versions, downloads, vulnerability counts, and other data surfaced in this year’s report. While Sonatype has direct access to primary data for Java, JavaScript, Python, .NET and other component formats, we also reference third-party data sources as documented.

Design of the Survey Questions

Used to Analyze Open Source

Component Use in Enterprises

Questions were designed to enable quantitative analysis. Most questions were built on a 7-point Likert scale measuring extent of agreement (“strongly agree” to “strongly disagree”) or time scales (e.g. “How frequently do you deploy to production?” with options such as “with every change,” “multiple times per day,” “multiple times per week,” “once per week,” etc.). Where there were multiple ways to ask about a particular attribute (e.g. “Job Satisfaction”), multiple questions were included and combined into a single dimension for analysis (e.g. “I am satisfied with my job,” “I would recommend this organization as a good place to work,” “I have the tools and resources I need to do my job,” etc.). When multiple questions were combined into a single measure, we verified that the question responses were strongly correlated and

used principal components analysis to perform the dimensionality reduction.

Independent Variables

Measured When Analyzing OSS

Component Use in Enterprises

In our survey of over 600 development professionals to assess how practices and outcomes related to their use of open source components, we measured the following factors to test their effects on the independent variables described above:

DEVELOPMENT PRACTICES Development philosophy: the general

DEVELOPMENT PRACTICES Component Use in Enterprises

OSS SUPPLIERS Development philosophy: the general

Development philosophy: the general philosophy of development practice used by your team on a spectrum from “waterfall” to “agile / DevOps”

BUILD, TEST, AND RELEASE Confidence in automated testing: To
what degree are you confident that when the automated tests pass the application will operate as intended in production.

BUILD, TEST, AND RELEASE Confidence in automated testing:

Deployment automation: to what degree are your application deployments (and configurations) automated.

(i.e., security issues are treated as normal defects).

⊲ For company-sponsored OSS Independent Variables

Static analysis tools: To what degree are the output of static analysis tools (e.g., Checkmarx, Coverity, Fortify, etc.) integrated into your daily development workflows.

Artifact repository centralization: To what degree can you centrally

To what degree can you centrally analyze all your deployed artifacts (e.g., executable binaries, Docker containers, infrastructure as code, etc.) for open source governance compliance.

which OSS is supported within the organization, as measured by the following:

OSS SUPPLIERS OSS selection criteria: What factors are considered when you decide whether to use an OSS component, specifically popularity, feature set, ease of integration, security history (e.g. have there been multiple high-risk CVEs), rate of fixes (frequency of security and bug fixes), OSS license, commercially available support, and foundation/corporate sponsorship.

⊲ For company-sponsored OSS projects, to what degree are external contributions allowed?

OSS enlightenment: The degree to

⊲ To what degree does your organization require that all internal modifications to open source components be contributed back (i.e., “pushed upstream”)?

OSS PHILOSOPHY

⊲ To what degree does your leadership support contributing back to open source components we use (e.g., engineering time, budget, conferences)?

Process to add OSS components: The degree to which you use a well-defined process to add new dependencies to an application (e.g., evaluate, approve, standardize, etc.).

ORGANIZATION AND POLICY

Centralization of asset management: The degree to which there is centralized

Centralization of asset management: The degree to which there is centralized tracking for every deployed application, its open source dependencies, and ability to contact the application team members.

The degree to which you enforce open source component governance (e.g., security, licensing) through your CI infrastructure.

dependencies scheduled as part of

Process to remove OSS components:

remove problematic dependencies.

Centralized OSS governance: The degree to which there is a centralized committee/group/team that is responsible for monitoring and enforcing open source component governance (e.g., security, licensing). OSS enforcement via automated CI: sonatype

Sonatype is the leader in software supply chain automation technology with more than 300 employees, over 1,000 enterprise customers, and is trusted by over 10 million software developers. Sonatype’s Nexus platform enables DevOps teams and developers to automatically integrate security at every stage of the modern development pipeline by combining in-depth component intelligence with real-time remediation guidance.

For more information, please visit Sonatype.com, or connect with us on Facebook, Twitter, or LinkedIn.

European Office

Headquarters 8161 Maple Lawn Blvd, Suite 250 Fulton, MD 20759 USA • 1.877.866.2836

APAC Office 5 Martin Place, Level 14 Sydney 2000, NSW Australia

Sonatype Inc. www.sonatype.com Copyright 2020 All Rights Reserved.