The quantitative research summarized below, covering over 7,000 repositories across nearly 100 countries, highlights some of the challenges with quality at modern development velocities, especially important for DevOpsSec practices. By leveraging automation in your repository manager, you can improve application quality and reduce unplanned work while lowering exposure to risk. While this practice supports DevOpsSec initiatives, at its core, it is simply a focus on “building quality in”.

(Written as a collaboration between me and Mike Hansen, Head of Products at Sonatype; Mike wrote the better parts.)

Innovation and Risk

Repository managers like Nexus, Artifactory and Archiva have been serving software components for development teams and their tooling for years now.  We can measure over 70,000 active repository manager installations across the globe today.  Use of a repository manager improves build performance and reliability while also providing a secure, private location to host open source, third party, and proprietary components needed across continuous delivery and DevOps tool chains.

Industry averages show software development organizations electively downloading over 240,000 components a year from public internet repositories.  While these components accelerate development and increase innovation, the vast majority of components go through little to no vetting process to determine if known security vulnerabilities, risky license types, or outdated/unsupported are being used.

Earlier this year, researchers found that 1 in 16 downloads includes a known security vulnerability. That accounts for over 15,000 elective downloads of components with known security vulnerabilities per organization, per year.

Why the Focus on Repository Managers?

Repository managers like Archiva, Artifactory, and Nexus have become a critical piece of the DevOps toolchains and are commonly used across the entire application lifecycle. It is effectively a binary parts warehouse for your applications. Given how integral this warehouse has become, software supply chain intelligence and automation can be used to significantly reduce unplanned work and eliminate avoidable risk.

To understand this opportunity, let’s take a quantitative look at the world’s software development ecosystem.

Global Visibility

Not long ago Sonatype introduced a free feature in its Nexus Repository Manager that is providing new data on the health of software supply chains.  This features, called Nexus RHC (Repository Health Check) – offers a simple way to get basic visibility into what OSS is flowing into development via your repository manager.  RHC provides a summary of components in the repository that have known security vulnerabilities and risky license types. Usage of this service is now significant, with over 30 million components across over 15,000 Nexus instances being analyzed every day across the globe.

There are also around a million developers using those repositories. With the law of big numbers working for us, we can gain a statistically valid view into the behaviors of the modern software development ecosystem. The findings are quite telling.

The Data

We analyzed over 7,000 repositories containing 500 or more software components using the RHC service during the past 90 days. These repositories are representative of those used by development teams in medium to large organizations. The question was straightforward: To what extent are bad things flowing in, causing downstream rework and creating avoidable risk scenarios?

In three months time, the average number of new vulnerabilities that flowed into the repositories analyzed was 69. That is an average of 23 vulnerabilities per month, which is a little more than one every weekday. If you only include higher risk vulnerabilities – those with a CVSS score of 5 or greater – the average number was 48, or about 16 per month. That is a little less than one every weekday. For a sense of the kind of vulnerabilities in this higher risk category, the Heartbleed bug had a score of 5, the lowest risk in this grouping.

These are not isolated instances or outliers. In the set of repositories analyzed, 98.3% consumed at least 1 vulnerable component and 97.7% consumed at least 1 higher risk (CVSS >= 5) component. In a period of just three months, nearly every repository consumed something the organization running it would rather not have, and that is only with respect to known vulnerabilities. There are plenty of other attributes that are worthy of consideration given availability of the corresponding data.

Whether or not security is of particular concern, the real point is that a lack of visibility and control combined with an abundance of supply has led to inferior quality parts routinely being used by the world’s development organizations, adding avoidable risk and slowing us down.

Going (Way) Beyond Security

The consumption of vulnerable components is a quantifiable problem that we used to illustrate the current challenges that every organization has with open source component selection. However, security vulnerabilities are only one of the many dimensions.

Others include architectural aspects such as rules covering component age and popularity, intellectual property coverage with rules related to open source licensing and technical debt controls associated with technology stack selection.  Introducing more formalized vetting processes coupled with software supply chain intelligence and automation will yield significant overall quality and efficiency benefits for an organization, allowing them to operate with a lower risk profile.

Yes, A Firewall for Repository Managers

In November, Sonatype became the first company to introduce a “firewall” for use with repository managers.  Called Nexus Firewall, the solution offers perimeter quality control for software development. Similar to a network firewall, it leverages rules you define that automatically shield an organization from unacceptable software components entering and another set for stopping them from exiting your application development. The basic concept looks like this:

In order to build quality in from the beginning, this solution represents a “shift (far) left” innovation.  The Nexus Firewall improves speed and reduce risk through the quarantine of components with known vulnerabilities. With a repository firewall in place, organizations can shield your application development from waste and risk by automatically blocking unacceptable software components inbound and preventing release of applications containing such components outbound.  1 in 16 downloads with known security vulnerabilities could be reduced to zero in 16.

The firewall also goes beyond blocking, providing organizations with the visibility and data needed to make ideal decisions for open source component selection early, significantly reducing waste related to rework and eliminating avoidable risk.

For DevOpsSec, Shift (Far) Left to Build Quality In

The consumption of these unacceptable or, at a minimum, inferior components is unnecessary.  For the vast majority of these vulnerable components, there are non-vulnerable alternatives ready to be used in their place. The challenge is that it has been historically difficult for developers to understand these risks and even harder to avoid them.  Few developers have countless hours to commit to researching vulnerabilities in the components they use and fewer more want to rely on colleagues in security, legal, or audit teams to spend hours or weeks to return with “go” or “no go” vetting decisions.

The volumes and velocity of component consumption we see today validates that any form of manual review process or automated workflows outside of the development teams will be quickly outpaced.  DevOps and Continuous Delivery processes need to rely on automatic intelligence to help developers select the highest quality components and avoid those with known risks.  When known security vulnerabilities and other risks are identified, software supply chain intelligence will be required to guide developers through the identification and selection of safer, higher quality components.

An example of the kind of information made available is shown below.

The information is divided into two areas. On the left side is component data, which includes details related to the component itself. To the right, there’s a graphical display of any security or license issues, as well as popularity data for each version of the component displayed. Selecting different versions updates this information accordingly, including whether or not a particular version passes the established criteria.

A Must for DevOpsSec: Quality at Velocity

Simply by enabling a repository firewall, organizations can immediately improve quality and reduce waste and exposure to risk. None of this comes at the expense of development speed.

The repository firewall automatically quarantines components that do not pass your rules preventing quality issues from entering the software being developed, immediately reducing risk and avoiding wasteful rework at some later point. Quality is improved, efficiencies are gained and risk is reduced, all through automation and all at the speed of modern development.

One of my favorite reference books is Continuous Delivery by Jez Humble and Dave Farley.  Let me share a few of their words to help remind us of the importance of innovations and practices that allow us to bring pain forward.  Jez and Dave remark:

“The techniques that we describe in this book, such as continuous integration, comprehensive automated testing, and automated deployment, are designed to catch defects as early in the delivery process as possible (an application of the principle “Bring Pain Forward”).  The next step is to fix them  A fire alarm is useless if everyone ignores it.  Delivery teams must be disciplined about fixing defects are soon as they are found.”

Well said Dave and Jez.  Well said.