DevOps’s best asset is swift and valuable change, which also may be its biggest risk factor. DevOps certainly challenges change management and security. We recently discussed these challenges and the solutions with Deena Coffman, managing director at BDO Consulting’s Technology Advisory Services practice. Here is part one of that exchange.
David Geer: What kinds of challenges does DevOps create for change management? Can you illustrate with an example?
Coffman: By combining the sysadmin and developer functions, DevOps can lend to “shortcutting” the change management process in the interest of expediency. It may also undercut the value of having a critical eye from both a distinct system administrator and a distinct development reviewer. DevOps is designed to move quickly, and while the pace of change is accelerating, speed is often an ingredient for a security incident. Much like having the same individual who approves invoices also issue a check, DevOps makes for a faster, but not always secure, process.
When the system of checks and balances can be purposefully circumvented, that privilege tends to be invoked more often than necessary. As a result, developers may bypass change management altogether far too frequently. The collaborative benefits from DevOps integrating typically siloed functions such as QA, software development, security and information technology operations are important, so long as they aren’t offset by compromises in quality and security that result from the reduction in the review and approval process.
The expedient nature of DevOps can also lead to error, given the absence of human experience involved in the evaluation process. Before DevOps, if a developer needed to change a setting on the web server, they would submit a change request and wait for approval. Under a DevOps structure, though, the developer simply scripts the change and commits it to the repository—automatically kicking off the build process. The process runs unit testing, integration testing, functional testing and regression testing against the new code, and then pushes the new code to the staging environment and finally, to production.
Geer: What kinds of challenges does DevOps create for security? Can you illustrate with an anecdote?
Coffman: The biggest security challenge in a DevOps environment is that speed becomes paramount, which reduces critical evaluation on security risks. Developers incentivized to quickly deploy new capabilities are naturally pushed toward productivity and efficiency, which often run counter to security. DevOps can skew that balance so that security is only lightly or sometimes insufficiently considered. Automated testing will test for known exposures, but a trained, experienced human may be able to anticipate a risk before it materializes.
Geer: How can flaws in change management in DevOps approaches lead to security issues?
Coffman: Flaws or inefficiencies in the change management process reinforce the views of those who are prone to bypassing what they perceive as tedious process requirements. If a change management process is too burdensome or seen as lacking value, people will avoid it at any opportunity—not just when an exception permits a bypass. Since technology professionals tend to have a bias against waste and inefficiency, introducing a process requirement that consumes time and effort without returning value has a high probability of triggering a wholesale negative reaction.
Geer: What kinds of things do DevOps shops and teams need to be doing to address change management so that they continue to meet the expectations of the business?
Coffman: When it comes to ensuring that change management meets business expectations, it really comes down to several key ingredients: flexibility, speed, efficiency and risk management.
DevOps teams should design and manage the change management system to be efficient, responsive, and flexible enough to accommodate cursory reviews for low risk scenarios and careful, in-depth reviews for other changes. Leadership needs to also set a tone of support for the change control process so it is not dismissed or marginalized. Additionally, engaging outside expertise and/or including additional quality and security measures for higher risk projects are recommended.
A risk review that identifies projects with security and privacy risk and distinguishing types of changes supports the automation decisions and builds a DevOps program that balances its benefits with risk management.
Stay tuned for part two.