More and more software teams are waking up to the fact that their CI/CD pipelines are prone to risk. These pipelines often interface with various toolsets and automation, which may be prone to misconfigurations and abuse. This is not to mention the potential for insecure code being pushed through this funnel and external factors that threaten to circumvent the pipeline.
In recent years, these risks produced some CI/CD tooling breaches—concerns that gained high-level attention recently after a joint CISA and the NSA alert about the importance of securing CI/CD pipelines. “This is something that moves the tectonic plates of the industry,” said Daniel Krivelevich, former founder of Cider Security and current CTO of AppSec at Palo Alto Networks.
I reconnected with Krivelevich, the ‘godfather of CI/CD security,’ to reevaluate the CI/CD threat landscape and understand how IT should respond. According to Krivelevich, we’re now seeing more maturity around CI/CD security, both in terms of recognizing that these risks exist as well as emerging frameworks to help engineering teams respond. He shared a general three-step framework organizations can use to secure the CI/CD pipeline and surrounding areas, which we’ll explore below.
The Evolving Threat Landscape
The traditional approach to application security (AppSec) used to be simpler. Defenders could implement more standard controls to make their code more secure, said Krivelevich. However, throughout the years, the software engineering process has transformed dramatically. It’s become faster, dynamic and reliant upon third-party frameworks. “The entropy of the engineering ecosystem is very high,” he said.
This paradigm shift has introduced a slew of new risks, which requires rethinking since most traditional security solutions deal solely with AppSec. “Software engineering is a big part of what makes the business tick,” said Daniel. Therefore, having a common language and understanding of this area is crucial to respond to evolving risks that threaten business operations.
A Three-Step Framework For CI/CD Security
Krivelevich is involved with creating the OWASP Top 10 CI/CD Security Risks; therefore, he has excellent insight into the types of vulnerabilities that most engineering practices and their CI/CD pipelines are facing. His three-step framework for CI/CD security, which Palo Alto Networks has applied in practice, looks to holistically secure not only the pipeline itself but also internal and external factors.
1. Security in the Pipeline (SIP)
SIP can be thought of as the traditional AppSec problem space. This involves ensuring developers are not shipping anything inherently insecure into production. Therefore, whatever code flows through the CI/CD pipeline must be vetted to ensure it has no security flaws or misconfigurations. Of course, this is easier said than done, given that teams are juggling more technologies and frameworks than ever, each of which has different knobs and requires unique security scanning capabilities.
2. Security of the Pipeline (SOP)
This area has to do with the security of the software delivery chain itself. Attackers may attempt to breach these pipelines by running malicious code within the pipeline or stealing credentials to CI/CD tools to harm the system. Poisoned dependencies, access management holes or insufficient credential hygiene could all be doorways to disrupting a CI/CD process. Therefore, SOP focuses on the security posture of the actual systems and tools that compromise the software delivery chain.
3. Security Around the Pipeline (SAP)
However, even if SIP and SOP are secure, what if hackers can bypass the pipeline altogether? That would render even a perfect set of controls null and void. Therefore, SAP is necessary to ensure that attackers can’t push code directly into production. It’s good to see the bigger picture so that everything that reaches production actually originates from the pipeline.
To deploy SAP effectively, Krivelevich recommended leveraging both detective and preventative measures. You need detective measures to issue alerts when anomalies occur, such as an artifact deployed by a human account instead of a service account. You also need preventative measures that set the proper branch protection rules across all repositories. For instance, this could be a rule that no single developer can publish to production unless approved by two reviewers.
The Role of AI in Software Security and CI/CD
By using both detective and preventative measures, you can detect when settings are disabled or abused and have better configurations across all stages of the software delivery chain. But what other tactics can be used to secure CI/CD? For example, does machine learning have any merit in this area?
Well, Krivelevich described AI as “both a blessing and a curse.” Attackers are already using AI to operate much faster and in a much more agile way. And unfortunately, he viewed the engineering ecosystem as the new path of least resistance. “I expect the power of AI will be used by sophisticated attackers who are motivated to get access to the ‘crown jewels’ by abusing the engineering ecosystem,” he said.
On the defensive side, Daniel doesn’t see a specific opportunity for deploying AI to protect CI/CD. That being said, there’s a lot of potential for security builders to utilize AI to advance their efforts and accelerate the pace at which they detect anomalies, which could help respond more quickly to emerging threats.
Final Thoughts on CI/CD Security
The recent chatter around CI/CD threats appears to be garnering more attention around securing this niche area of agile software development. To thwart CI/CD attacks, it will be necessary to assess the settings of systems, understand their technical natures and analyze them against an attacker’s perspective, Krivelevich explained. Therefore, understanding what realistic attacks can be carried out will help build measures to correct these gaps.
In doing so, the SIP, SOP and SAP descriptions serve as a solid reminder of what needs to be addressed when attempting to secure the software delivery process and the supply chain at large. Code that goes through the pipeline needs to be free from vulnerabilities, and the channel itself must be vetted for gaps that could be abused. We also need to consider the possible existential threats if the entire process can be undermined.
