If you’ve ever seen people run a relay race, you’ll notice that most of the risk happens when the baton is handed from one runner to the next. In IT processes, there is also dramatically increased risk during handoffs. Whether you’ve moving from one process phase to another, one “owner” to another, or one environment to another the risk is greatest during the phase transition.
Security can play a strong role during these handoffs, whether you’re using DevOps or not. Here are some principles that can help safeguard the handoffs and reduce the risk in the operational relay race.
They say too many cooks spoil the broth and that can apply to IT processes, as well. Limit the number of people who can directly make changes in production and you have a shot at increasing security and defending availability by reducing random acts of deployment, as well as change collisions and mystery changes that just show up in production. This is a key anchor for creating predictability and reducing change collisions.
Create defensible “choke points”
Reducing the number of ways things can be deployed into production will help you defend the integrity of the production environment. This doesn’t necessarily mean preventing changes, it is more about controlling how changes enter the environment. Focus on creating trustworthy repositories, deployment sources, automation tools, and service accounts that reduce the options for how deploys happen. When you define clear swim lanes and you know what the expected path of deployment looks like, it is easier to detect unexpected activities and actions.
Use choke points to enforce acceptance criteria to ensure that you are ready to take responsibility for the code, systems, and applications that being handed to you.
This approach allows great freedom in how work gets done in autonomous teams, while enforcing more consistency through the release management and transition to Ops phases of the life cycle.
Think like an air traffic controller
When dealing with multiple streams of work that need to come together and interoperate in production, I think a mental model of an air traffic controller works well. Airports allow a bunch of independent operators to come together in a shared environment in an orderly fashion, with minimal collisions and predictable throughput, and the air traffic controller plays a key role in making that happen. In the IT context, this means leaning on people who have a broader perspective than any individual team so they can look at the big picture and coordinate activities holistically.
This “zoomed out” perspective is critical in identifying risky activities and gaps, understanding dependencies that can break your security model, and stepping in to help mitigate dangerous conditions. This approach also makes it easier to work with third-party providers, since you can focus on their results (outputs and handoffs) rather than getting caught up in the minutiae of their day-to-day activities.
Monitor continuously to identify “trust gaps” and outliers
DevOps is no stranger to continuous monitoring and instrumentation. From a security perspective, continuous monitoring is about establishing a set of controls that allows you to quickly identify conditions that increase risk, and the elements above help you do that job more effectively. For example:
- Access limits and choke points enable you to quickly detect when someone has gone around your process and made changes directly in the production environment.
- These elements also allow you to keep tabs on third-party contractors and providers to ensure that they play by your rules, and you know what they are up to all the time.
- Defensible choke points enable you to sound the alarm if someone is tampering with your source repositories, automation scripts, or security monitoring components (such as disabling logging).
- Study your indicators to find dark spots in your telemetry around handoffs, and ask how an attacker (internally or externally) could exploit any weaknesses in your handoffs.
These are just a few examples of how these concepts can be applied. When you recognize that handoffs are risky, you can quickly see how security can play a role in mitigating those risks and helping defend the reliability, safety, and trust of your production environment.