A Deep Dive into Open Source Security Measures | Sonatype

CHAPTER 2 Open Source Security Practices

In 2020, an OpenSSF project started collecting information on software security practices employed by open source projects. This substantially increased transparency of open source development by centrally tracking the usage of commonly accepted best practices.

In a previous year's State of the Software Supply Chain report, we analyzed the extent to which higher scores on these checks are associated with better security outcomes and found them to be highly predictive of vulnerability status. In this year's report, we delve into the state of adoption of Security Scorecard best practices overall and also by ecosystem (focusing on Java and JavaScript). We look at checks today as well as over time, looking back at the prior year to see both how community practices have evolved and how the checks themselves have evolved over time.

Our dataset consists of Scorecard checks from 2022-06-27 and 2023-07-03, retrieved from the Scorecard BigQuery database. Each check is scored from 0 to 10. If the check fails or there is lack of evidence for a check, the Scorecard project assigns it a score of -1. We mapped these -1 values to 0 for our analysis.

The Checks

BINARY ARTIFACTS

Measures whether any binary artifacts are checked into the repository.

BRANCH PROTECTION

Checks whether the project uses branch protection on its default and release branches to prevent maintainers from circumventing workflows like CI tests or code review when pushing changes.

CII BEST PRACTICES

Checks whether a project has obtained an OpenSSF (formerly CII) Best Practices Badge.

CODE REVIEW

Checks whether recent code changes have been peer-reviewed before being merged.

DANGEROUS WORKFLOW

Checks whether the project avoids dangerous coding patterns in GitHub Action workflows.

DEPENDENCY UPDATE TOOL

Checks whether the project is using tools to help update its dependencies.

FUZZING

Checks whether the project is using fuzzing tools.

LICENSE

Checks that the project includes a license file.

MAINTAINED

Scores the project based on the amount of commit and issue tracking activity it has had over the last 90 days.

PACKAGING

Checks whether the project builds and publishes official packages from the CI/CD pipeline.

PINNED DEPENDENCIES

Checks whether project dependencies (including dependencies for GitHub Actions and Docker files) are pinned to specific versions.

SECURITY POLICY

Checks that the project includes a security policy.

SIGNED RELEASES

Checks that the project cryptographically signs releases.

TOKEN PERMISSIONS

Evaluates whether the project's automated workflow tokens follow the principle of least privilege.

The state of Scorecard check scores

In last year’s report, we conducted an analysis of the connection between Scorecard checks and known vulnerabilities. One outcome of this analysis was information on which checks are most closely associated with better vulnerability status. Figure 2.2 summarizes this information. Code Review (e.g. requiring review of pull requests before merging) was the most important practice, followed by Binaries (not checking binary data into the repository), Dependencies Pinned (pinning dependencies to specific versions), and Branch Protection (preventing direct pushes to the main branch).

FIGURE 2.1. ELEMENTS MOST USEFUL FOR IDENTIFYING VULNERABLE PROJECTS

The figure below shows how widely the Scorecard best practices have been adopted. The chart presents the average value for each check across our 2023 dataset. Note that Code Review, which we found last year to be the practice most highly associated with good security outcomes, is not widely practiced in general, with an average score of less than 1. Roughly 19% of projects score greater than 0 on Code Review, and only 2% receive the full score of 10 (indicating that all code changes have undergone review). Branch Protection, which supports code review by requiring changes to go through the review process, also scores low (just 0.24 on average). The Vulnerabilities check measures whether a project has open, unpatched vulnerabilities, either associated with the project itself or with its dependencies. The Vulnerabilities score starts at 10 and is decreased by 1 for each open vulnerability with a minimum score of 0 (e.g. a project with 10 open vulnerabilities and a project with 15 open vulnerabilities both score 0). This means that the average score of 8.9 indicates that projects on average have at least 1 open vulnerability. Of course not every project is vulnerable: just over 14% of projects have a Vulnerabilities score of less than 10 (indicating they have at least one open vulnerability). If we look at just vulnerable projects, we find that on the date our 2023 data was collected they had, on average, at least 7.7 open vulnerabilities per project.

FIGURE 2.2. SCORECARD CHECK AVERAGES, ALL PROJECTS

Another important low-scoring check is Maintained, which checks whether a project is being actively maintained, either via regular code commits or timely responses to issues. Maintenance is generally a prerequisite for security patches. On average, projects score about 0.66 on the Maintained check. Out of 1,176,407 total projects, just 118,028 of them (~11%) have a Maintained score greater than 0.

Looking at the data by ecosystem (shown in the figure below), we see that Java projects score higher on average than JavaScript projects for a number of security-relevant checks.

Java projects score on average:

FIGURE 2.3. SCORECARD CHECK AVERAGES, JAVA PROJECTS

Java projects also score 10x higher on Fuzzing, though very few projects in either ecosystem take advantage of fuzzing technology. Rates of vulnerabilities are comparable, with 15% of Java projects and 16% of JavaScript projects having unpatched vulnerabilities at the time of scan. Note that this is a measure of whether there are vulnerabilities that apply to the development branch of a project. This does not indicate whether the latest release of the project has a vulnerability, nor whether the recorded vulnerabilities will persist into the next release.

The importance of maintenance

The figure below shows how the average scores change when we focus on maintained projects only (projects with a Maintained score greater than zero). There are several notable differences.

Maintained projects are:

FIGURE 2.4. SCORECARD CHECK AVERAGES, MAINTAINED PROJECTS

Maintained projects also score better on Vulnerabilities and have slightly lower rates of vulnerability.

Year-over-year differences

We also looked at the data from one year ago to see how Scorecard practices have changed. Surprisingly, we find that there has not been a steady improvement in Scorecard scores. Instead, we see decreases in several key metrics. In particular, the number of maintained projects has decreased overall and also within each ecosystem. Overall, there are 24,104 projects that were maintained one year ago but no longer qualify as maintained. This represents 18.6% of the maintained projects. Put another way, if you picked a random maintained project in 2022, there would be a 18.6% chance that this project is no longer maintained one year later.

Overall, these changes highlight the importance of tracking the health of dependencies over time. Just as new projects are constantly being created, projects are also constantly reaching end of life status. Identifying these changes is an important part of maintaining a healthy set of application dependencies.

We have also seen an overall decrease in the amount of code review. The number of projects that had any Code Review decreased by over 15%. The decline was slightly less for Java (11%) than for JavaScript (17%). Even when just looking at maintained projects, there was a decrease of 8.6% in rates of code review. Comparing trends in Scorecard checks over time is complicated by the fact that the OpenSSF Scorecard team does periodically change the checks in ways that can affect scoring. However, we reviewed the changes made to the Code Review check and, on aggregate, they appear to make the check easier to pass (e.g. by treating a PR merged by someone other than the contributor as a review). If this data indeed reflects a decline in the amount of code review occurring in otherwise well-maintained projects then this is a worrying trend. We recommend ensuring that any open source libraries critical to business applications are practicing code review on every pull request to the main branch.

Conclusion

Overall, these results provide guidance for open source users when choosing dependencies and important areas of focus for open source advocates and maintainers. The value of choosing well-maintained projects stands out, as these projects tend to be more diligent about a number of important software security best practices. The fact that 18.6% of projects stopped being maintained in the last year highlights the need to not only choose good dependencies, but monitor those dependencies for changes in their quality.