A DevOps solution to the DevOps problem
Intel IT’s Enterprise landscape is highly complex and diverse. The Enterprise Change Management team has the responsibility of managing the changes in this landscape, which is of the order of hundreds of changes every week. On one hand, we had to make sure we keep the change management processes stringent so that defects/major incidents did not happen in the production environment, leading to productivity loss, shipment delays or cost impacts. On the other, there is a need to assist the capability teams to land changes into production faster than ever to meet their dynamic business demands.
A challenge that many global IT organizations face is, How do we adhere to ITIL processes while teams transform into agile/DevOps way of working? In other words, how do we manage the two seemingly conflicting goals from two different IT organizations, one focusing on the agility of the development and measuring the success of the projects by their throughput time and the other measuring the success by the quality?
In the IT enterprise, the quality is measured by the number of change induced incidents in production—incidents that are caused due to change(s) landing in production. Our team had a challenging goal of keeping it to zero! The year 2017 started with this million-dollar (or is it worth more than that?) question: How do we meet this goal while not impacting the speed of the agile teams? We came up with an agile solution. A DevOps solution for the DevOps problem sounds logical, right?
We formed a scrum team to address this problem with the theme, “Simplify and automate, keeping the focus on quality, velocity and productivity.” The scrum team had change management process owners and the representatives from all the development technologies. The chart below summarizes the steps followed:
Simplify and Automate, Keeping the Focus on Quality, Velocity and Productivity
The first sprint was all about getting the alignment. As much as the number of teams that follow agile, as many definitions of agile exist! The focus was to demystify the myths about agile as well as ITIL change management processes. At a glance, processes look incompatible with Agile, but the goals are the same. Both aim to add business value incrementally in a continuous manner without disrupting the operations. They are not contradictory but complimentary.
Subsequent to this alignment, the second sprint went to the details of the enterprise change management process that’s in place. The process was broken down into the lowest level of details based on the 5W and 2H’s (who, why, what, when, where, how, how much). For example, there are process requirements from legal standpoint that are non-negotiable. There are certain processes required to ensure architectural compliance. We also acknowledged and aligned on some of the challenges and the time taken for process adherence. Multiple discussions happened in this space and it helped all of the teams to come to a common understanding on why these process requirements exist. Process swim-lane categorizing all of these aspects was internalized as part of this sprint.
Change Management Process Swim Lane
The third sprint got things into action mode. Each of those process steps was identified for potential simplification and automation. They were further classified as short-term and long-term based on the automation efforts on continuous integration/continuous delivery (CI/CD) that were underway. The development teams were shown the flexibility and empowered to meet the process need by way of automation rather than completing a manual checklist for every change. This way of policy and rule-based automation helps in a longer way in integrating ITIL Change Management and DevOps.
All along these three sprints, process improvements were rolled out. Several automation efforts were launched. Objective risk assessment enabled us to identify changes that required advance communication to all segments. Multiple gate approvals were reduced into only two—one when the development starts and later before it goes to production. Test cases and results from the CI/CD tools were used effectively to replace documentation.
Process refresher and engagement sessions were held across the organization to share the improvements, gather feedback and incorporate them into further sprints. This helped to gain the confidence of various teams and made the journey further better.
Now the focus is on continuous improvement, which is to implement all those action items identified and also keep them updated as other factors change in the internal and external environments. Engagement between theAgile/DevOps teams and the change management teams are getting refined based on the scrum development models. The success of ITIL Change Management and DevOps integration relies on calling out the agile process followed for the project and having the right engagements at the right points in time. The below picture depicts the engagement in a simplistic manner:
Scrum and Change Management Intersection
The efforts yielded fruits and the organization met the goal. The number of change induced major incidents was ZERO for the Enterprise Segment for the whole of 2017. We need to remember that these process improvements were done without impacting anything underway; it was akin to servicing a flying aircraft in the middle of the sky. The feather in the cap was the positive feedback from the ERP vendor: “The change control process at Intel has become very agile in recent years, with transports to production now occurring nearly every day of the year. … In general, we do not consider such frequent changes to Production to be desirable due to the risks involved; however, Intel appears to be managing the process well as the quality of transports is closely monitored.”
References:
- ITIL® Service Management v3
- The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations by Gene Kim, Jez Humble, Patrick Debois & John Willis
This article was co-authored by Ramani Subramanian