Management guru Peter Drucker once said, “If you can’t measure it, you can’t improve it.” I would take a step back and say, If you can’t measure it, you can’t get it off the ground. Many companies that I work with are amid digital transformation initiatives, such as the adoption of DevOps and continuous delivery. For these initiatives to gain momentum and acceptance within their organizations, they must be able to demonstrate quick time to value or risk being sidelined. And then, these organizations must be able to actualize their transformations and improve their processes—both enabled by the right metrics.
Indeed, one of the core pillars of DevOps culture—going all the way back to the first DevOpsDays event in 2010—is measurement. It is the M in the CAMS (Culture, Automation, Measurement, Sharing) acronym coined by Damon Edwards and John Willis.
Glaring Gap in Continuous Delivery Metric Definitions
Despite the prominent placement of measurement in the DevOps world, there is not really a lot of guidance about metrics in the common literature. For example, the original “Continuous Delivery” book by Jez Humble and David Farley devotes maybe a dozen pages out of 500 to it. And the recent “DevOps Handbook” by Gene Kim, et al is thin on the topic as well.
The coverage gap around measurement and metrics is driven primarily by the fact that the DevOps space is new and there are few standards around which to build an easily documented metrics structure. That leaves practitioners with the challenge of creating something they can use pragmatically within their own organizations. Anyone building a system of measurement for their organization must have a reasonable understanding of metrics as a discipline and a good understanding of the processes they intend to measure.
Back to Basics on Measurement
There are two types of metrics that I discussed in a recent session at CA World ’16 that are important when measuring a complex enterprise process.
- Result metrics are concerned with the cumulative outcome of a set of activities. For example, a company’s revenue or net income would be common, classic result metrics.
- Performance metrics measure how well individual activities are working. These often are more difficult to define, as they can vary by businesses and teams and be obscured by silos.
Performance and result metrics are very closely linked: Performance metrics measure the effectiveness of the activities that drive the result metrics. For example, measuring the average number of deals closed per salesperson per period would be one measure of the effectiveness of a sales organization (a performance metric) which, in turn, would be a factor in driving revenue (a result metric).
While DevOps literature generally does not provide a lot of detail about metrics, it does provide advice on key result metrics. The primary one is lead time. Lead time is a powerful measurement that originated in Lean manufacturing. It originally dealt with how long it took a company to deliver a product to a customer after receiving an order. In DevOps terms, lead time summarizes the amount of time it takes to deliver one single change or feature from business idea through to running software in Production.
Lead time is calculated by aggregating the work time (referred to as cycle time) on a change plus the time when the change is in progress, but not being worked on (referred to as wait time). The formula that LEAD TIME = CYCLE TIME + WAIT TIME is a powerful tool for understanding the result metric level; however, the challenge in most enterprises is figuring out a workable system of supporting performance metrics.
Value Stream Mapping to Uncover Performance Metrics
You’ll find that the result metrics part of the effort—calculating the lead time for an application change—is relatively straightforward even when corporate structures can obscure exactly when work (cycle time) is happening versus when things are waiting (wait time). The Lean manufacturing world provides techniques for analyzing a flow.
A common technique is called value stream mapping (VSM). VSM provides guidance for breaking down a complex process into identifiable “steps” to provide measurement points throughout the process for cycle time and wait time. The measurement points then provide milestones for tracking the progress of the single change or feature from business idea through to production use.
For example, in an enterprise application development situation, typical application delivery pipelines provide several logical points that can be used as steps for VSM. Using those points, it is simple to measure linear time through the process. The common logical measurement points are:
- Assembly (aka Development): the time required to assemble a configuration package (code, database, infrastructure, and network) that the team declares as an instance of the application including the change
- Time Spent at Each Pipeline Phase: the time spent in various QA environments, such as Integration, Functional, Performance, UAT, etc., and to finally put the configuration package into the Production environment.
- Time Spent on Promotion Decisions: the time spent waiting for a decision to promote a given version of the configuration package to a higher environment or to reject it at the end of that validation phase.
The performance metrics that determine how efficiently cycle time and wait time are utilized, however, are significantly less obvious. Software development is iterative by nature. That means that individual steps in the value stream will be repeated many times and the cycle and wait time metrics ultimately will obscure the quality of what is happening at the measurement points due to natural tendencies to aggregate and average the values.
Performance Metrics for Continuous Delivery
Establishing the performance metrics for a continuous delivery pipeline requires a bit of analysis. The performance metrics must be geared to understanding how to reduce the cycle time per repetition and the overall number of repetitions at each measurement point in the delivery pipeline. Here is a pragmatic, two-part pattern you can follow:
The first part is to break down the iterative pieces of the pipeline into logical ‘mini’ value streams. For example, each QA environment must be provisioned, configured, have changes deployed, and then validation activities occur. Those steps can provide measurement points and, more interestingly, can be used symmetrically in all environments in the pipeline enabling questions such as, “Does one take a disproportionate amount of time relative to the others?”
The second part is to look at the qualitative items that drive either long repetitions or increase the number of repetitions. For example, how many environments does a configuration package go through before rejection? More can actually be more expensive, slower and likely to be indicative of requirements problems, while fewer may indicate code quality problems. Another example might be smoke test failure rate in each environment. Variance in one environment could indicate problems in either the configuration management or deployment automation, but would require repetition of the deployment or manual correction. Finding that problem and fixing it ensures better cycle time at that step and stage, and serves to keep lead time down.
End Game: Demonstrate Business Value
It may be a very long time before the DevOps movement reaches consensus on a system of metrics. The variability in software environments and industry needs means that it is unlikely for one size to fit all. However, with a pragmatic approach and some lessons from the Lean manufacturing world, it is possible to produce a workable and effective system of metrics for continuous delivery pipelines in the enterprise.
Vendors, such as my employer, CA Technologies, are helping by building more integrated systems of pipeline management tools to help shift from merely reporting metrics to applying them with modern analytics-driven techniques. The dividend of these approaches and tool improvements is that it is getting easier to gain a clear understanding of your process and to demonstrate to the business that the DevOps initiative is, in fact, worth the effort and investment.