Milk spoils. Iron rusts. And in software, good code goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.
Your 5,100 Binaries Went Bad
There is no way to prevent software from “going bad”. As with all products, bugs and defects,are bound to happen at some point. No one and no code is immune from these issues. But who’s looking for the 5,100 software components in your organization that went bad last year (meaning new security vulnerabilities were discovered in them)? It’s all too likely, no one.
Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt. How you ask?
Here is a summary of the their average debt:
Source: 2015 State of the Software Supply Chain Report
The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (“bad”) before they were downloaded.
The remaining 34% (approx. 5,100) might have actually been “good” components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.
Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered — about 50 new CVEs are announced daily — means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.
We’re Not Alone
Software development is not the only industry facing such issues of things good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.
The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.
In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version — dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to “recall” the impacted software or distribute newer versions of the application are hindered.
Did he recommend a “Bomb”?
A Bill of Materials (BOM; pronounced “bomb”) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product. According to Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.”
In Part 2 of this blog, “The Software BOM Squad“, I will introduce the importance of automating the creation of a “software bill of materials“ (Software BOM) to take inventory of the third party and open source components used to build an application. When software components age more like milk than wine, a software bill of materials can help you quickly identify where your 5,100 “bad” component were used.
