Human Error

We’ve all fat-fingered configurations and code before. Usually we catch them, but once in a while they sneak into production and wreak havoc on security. A number of “big names” have been caught in this situation, where a simple typo introduced a security risk. Often these occur because we’re so familiar with what we’re typing that we see what we expect to see, rather than what we actually typed. We’ve entered that command a hundred times, after all; we know what it’s supposed to look like and we aren’t as careful as we (likely) should be because it’s Friday and it’s almost 5 o’clock. Whatever the reason, we’ve done it, and it’s the repetitious nature of the beast that actually ends up biting us in the proverbial derriere.

Codifying those commands into scripts or, even better, into templates means once it’s right, it’s right. The inescapable reality that we will fat-finger something eventually is reduced to virtually nil because we aren’t typing it in over and over and over, each time with less and less attention. The reliance on codification, on templates and APIs and scripts as a means to automate tasks, is not just about speed but also repeatability. And accurate repeatability at that. That’s the kind of assurance you need to prevent “human error” from exposing apps—and the business—to a wide range of risks.

To reduce risk from human error via DevOps you can:

Use templates to standardize common service configurations;
Automate common tasks to avoid simple typographical errors; and
Read twice, execute once.

Lack of Process

This is probably my favorite risk (and not just because it lets me talk up Six Sigma and some really hairy mathematical equations) because it encompasses such a broad category of problems. First, there’s the fact that there’s almost no review of the scripts that folks already use to configure, change, shut down and start up services across the production network. Don’t let anyone tell you they don’t use scripts to eliminate the yak shaving that exists in networking and infrastructure, too. They do. But they aren’t necessarily reviewed, they are certainly aren’t versioned like the code artifacts they are, and they rarely are reused. Everyone has their “own” favorite language and scripts, and they are the Frank Sinatra of IT: They do it their own way. (Yes, I know you’re probably too young to get that reference, but until your musical icons have a better one, that’s the one I’m using because my generation didn’t have a better one, either.)

The other problem is simply there’s no governed process. It’s tribal knowledge. Bob does X and then Alice pushes Y and someone from Team Rocket* pushes button to make it go live. What can happen—and sometimes does—is one step is simply overlooked. It isn’t that there is no process, it’s that there’s no real standardization and no overarching governing process. There’s no one coordinating between Bob and Alice and Team Rocket to ensure the process goes smoothly. The thing is that the app might go live and actually work in production minus a few services, but if those services are related to security in any way (such as locking down a port), then your lackadaisical approach to the deployment process just put the business at risk.

To reduce risk from lack of process:

Define the deployment process, clearly. Understand prerequisites and dependencies and eliminate redundancies or unnecessary steps.
Move toward the use of orchestration as the ultimate executor of the deployment process, employing manual steps only when necessary.
Review and manage any scripts used to assist in the process.

External Threats

At first glance this one seems to be the least likely candidate for being addressed with DevOps. Given that malware and multi-layered DDoS attacks are the most existential threats to business today, that’s understandable. There are entire classes of vulnerabilities that only can be detected manually, by developers or experts reviewing the code. There’s DevOps in that, but it doesn’t really extend to production, where risks becomes reality when it’s exploited.

More extensive testing, and development of web app security policies during development that can then be deployed in production is one way that DevOps can reduce risk. Adopting a DevOps approach to developing those policies—and treating them like code, too—provides a faster, likely more thorough policy that does a better job overall of preventing the existential threats from being all-too-real nightmares.

To reduce the risk of threats becoming reality:

Shift web app security policy development and testing left, into the app dev life cycle.
Treat web app security policies like code. Review and standardize.
Test often, even in production. Automate using technology such as dynamic application security testing (DAST) and when possible, integrate results into dev life cycle for faster remediation that reduces risk earlier.

There is no technology, methodology or approach to IT security that will completely eliminate risk short of a complete power outage. Given that’s not a viable option, the best approach is one that reduces risk to acceptable levels (where acceptable is defined by your business’s tolerance, regulatory requirements and how much your MBO is based on not being hacked this quarter). DevOps and security are a good fit, as the former can help standardize, codify, automate and expand security practices across production and dev/test.

So get out there, and put some DevOps in your security. And while you’re at it, put security in your DevOps.

* Yes, that’s a Pokémon reference. I figured I needed something more modern after tossing Frank Sinatra in there. If you got both, give yourself +5 geek points.