What Quality Metrics Matter Most for DevOps?

The way we develop and deliver software has changed dramatically in the past five years—but the metrics we use to measure quality remain largely the same. Despite seismic shifts in business expectations, development methodologies, system architectures and team structures, most organizations still rely on quality metrics that were designed for a much different era.

Every other aspect of application delivery has been scrutinized and optimized as we transform our processes for DevOps. Why not put quality metrics under the microscope as well?

Are metrics such as number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure that the steady stream of updates don’t undermine the very user experience we’re working so hard to enhance?

To provide the DevOps community an objective perspective on what quality metrics are most critical for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in a new eBook, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts.”

The goal was to analyze how DevOps leaders measured and valued 75 quality metrics (selected by Forrester), then identify which metrics matter most for DevOps success. Here’s a look at the process:

Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
Learn what quality metrics those experts actually measure and how valuable they rate each metric they regularly measure.
Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

The 75 DevOps quality metrics were divided into four categories:

Build
Functional validation
Integration testing
End-to-end regression testing

For each category of quality metrics, we came up with a heat map showing usage versus value rankings. For example, here is the heat map for the Build category metrics.

We also plotted the data for each metric into a quadrant with four sections:

Value Added: Metrics that are used frequently by DevOps experts and consistently rated as valuable by the organizations that measure them.
Hidden Gem: Metrics that are not used frequently by DevOps experts but are consistently rated as valuable by the organizations that measure them.
Overrated: Metrics that are used frequently by DevOps experts but not rated as valuable by the organizations that measure them.
Distraction: Metrics that are not used frequently by DevOps experts and not rated as valuable by the organizations that measure them.

For example, here is the quadrant for Build category metrics:

The eBook provides both heat maps and quadrants for all four categories, a quick look at each of the 75 metrics per-category and overall analyses, as well as a few fun lists.

Here’s a preview:

Hidden Gems

The following metrics are not commonly used (even among DevOps experts), but are ranked as extremely valuable by the select teams who actually measure them: