DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Survey Surfaces Application Modernization Challenges
  • Dylibso Releases Tool for Tracking and Validating Wasm Modules
  • Data APIs: Realizing the Future of Data Warehousing
  • GraphQL Documentation Generators: How They Work and Why They Matter
  • Perceptions of Reality

Home » Blogs » Enterprise DevOps » What Quality Metrics Matter Most for DevOps?

What Quality Metrics Matter Most for DevOps?

By: Cynthia Dunlop on July 12, 2019 Leave a Comment

The way we develop and deliver software has changed dramatically in the past five years—but the metrics we use to measure quality remain largely the same. Despite seismic shifts in business expectations, development methodologies, system architectures and team structures, most organizations still rely on quality metrics that were designed for a much different era.

Recent Posts By Cynthia Dunlop
  • CXOs Overestimating DevOps and Continuous Testing Maturity
  • Continuous Testing in the ‘2018 Accelerate State of DevOps’ Report
  • Continuous Testing Insights from 2018 DevTest Research
More from Cynthia Dunlop
Related Posts
  • What Quality Metrics Matter Most for DevOps?
  • Relevant Continuous Testing: The Primary Key to DevOps
  • Meeting Challenges in incorporating automated testing throughout the DevOps cycle
    Related Categories
  • Blogs
  • DevOps Culture
  • DevOps Practice
  • Enterprise DevOps
    Related Topics
  • API tests
  • application delivery
  • automated tests
  • code coverage
  • DevOps leaders
  • DevOps metrics
  • DevOps research
  • Forrester research
Show more
Show less

Every other aspect of application delivery has been scrutinized and optimized as we transform our processes for DevOps. Why not put quality metrics under the microscope as well?

Are metrics such as number of automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure that the steady stream of updates don’t undermine the very user experience we’re working so hard to enhance?

To provide the DevOps community an objective perspective on what quality metrics are most critical for DevOps success, Tricentis commissioned Forrester to research the topic. The results are published in a new eBook, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics—Ranked by Industry Experts.”

The goal was to analyze how DevOps leaders measured and valued 75 quality metrics (selected by Forrester), then identify which metrics matter most for DevOps success. Here’s a look at the process:

  1. Survey 603 global enterprise leaders responsible for their firms’ DevOps strategies.
  2. From that sample, identify the firms with mature and successful DevOps adoptions (157 met Forrester’s criteria for this distinction).
  3. Learn what quality metrics those experts actually measure and how valuable they rate each metric they regularly measure.
  4. Use those findings to rate and rank each metric’s usage (how often experts use the metric) and value (how highly experts value the metric).
  5. Compare the DevOps experts’ quality metric usage versus that of DevOps laggards. If there was a significant discrepancy, the metric is considered a DevOps differentiator.

The 75 DevOps quality metrics were divided into four categories:

  • Build
  • Functional validation
  • Integration testing
  • End-to-end regression testing

For each category of quality metrics, we came up with a heat map showing usage versus value rankings. For example, here is the heat map for the Build category metrics.

We also plotted the data for each metric into a quadrant with four sections:

  • Value Added: Metrics that are used frequently by DevOps experts and consistently rated as valuable by the organizations that measure them.
  • Hidden Gem: Metrics that are not used frequently by DevOps experts but are consistently rated as valuable by the organizations that measure them.
  • Overrated: Metrics that are used frequently by DevOps experts but not rated as valuable by the organizations that measure them.
  • Distraction: Metrics that are not used frequently by DevOps experts and not rated as valuable by the organizations that measure them.

For example, here is the quadrant for Build category metrics:

The eBook provides both heat maps and quadrants for all four categories, a quick look at each of the 75 metrics per-category and overall analyses, as well as a few fun lists.

Here’s a preview:

Hidden Gems

The following metrics are not commonly used (even among DevOps experts), but are ranked as extremely valuable by the select teams who actually measure them:

  1. New defects (IT)
  2. Critical defects (FV)
  3. Automated tests prioritized by risk (Build)
  4. Code coverage (Build)
  5. Test cases executed (Build)
  6. Static analysis results (Build)
  7. Variance from baselines of percent of test cases passed (E2E)
  8. Release readiness (E2E)

Top DevOps Differentiators

DevOps experts/leaders measure the following metrics significantly more than DevOps laggards measure them:

  1. Automated tests prioritized by risk (Build)
  2. Percent of automated E2E test cases (E2E)
  3. Risk coverage (IT)
  4. Release readiness (FV, IT, E2E)
  5. Test efficiency (FV and IT)
  6. Requirements covered by tests (Build, FV, IT, E2E)
  7. Test case coverage (Build, E2E)
  8. Static analysis results (Build)
  9. Variance from baseline of percent of test cases passed (E2E)
  10. Test effectiveness (FV, IT, E2E)

Most Used by DevOps Experts

The following metrics are the most frequently used (overall) by DevOps experts/leaders:

  1. Test case coverage (E2E)
  2. Pass/fail rate (FV)
  3. API pass/fail rate (IT)
  4. Number of tests executed (E2E)
  5. API bug density (IT)
  6. Requirements covered by tests (FV)
  7. Requirements covered by tests (E2E)
  8. Blocked test cases (FV)
  9. Percent of automated E2E test cases (E2E)
  10. Successful code builds (build)

Most Valued by DevOps Experts

The following metrics are the most highly valued (overall) by DevOps experts/leaders:

  1. Requirements covered by API tests (IT)
  2. Percent of automated E2E tests (E2E)
  3. Requirements covered by tests (E2E)
  4. Requirements covered by tests (FV)
  5. Count of critical functional defects (FV)
  6. Total number of defects discovered in test (E2E)
  7. Number of test cases executed (E2E)
  8. Pass fail rate (FV)
  9. New API defects found (IT)
  10. Automated tests prioritized by risk (build)

— Cynthia Dunlop

Filed Under: Blogs, DevOps Culture, DevOps Practice, Enterprise DevOps Tagged With: API tests, application delivery, automated tests, code coverage, DevOps leaders, DevOps metrics, DevOps research, Forrester research

« DevOps Is Killing the ITOps Engineer (and That’s Bad)
AWS Advances DevOps Agenda at Summit Event »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Build Securely by Default With Harness And AWS
Tuesday, March 28, 2023 - 1:00 pm EDT
Accelerate Software Development Flow with Value Stream Management
Wednesday, March 29, 2023 - 1:00 pm EDT
Cloud-Native Developer Tools: What's on the Horizon?
Thursday, March 30, 2023 - 1:00 pm EDT

Sponsored Content

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Practical Approaches to Long-Term Cloud-Native Security

December 5, 2019 | Chris Tozzi

Latest from DevOps.com

Survey Surfaces Application Modernization Challenges
March 23, 2023 | Mike Vizard
Dylibso Releases Tool for Tracking and Validating Wasm Modules
March 23, 2023 | Mike Vizard
Data APIs: Realizing the Future of Data Warehousing
March 23, 2023 | Tanmai Gopal
GraphQL Documentation Generators: How They Work and Why They Matter
March 23, 2023 | Gilad David Maayan
Postman Releases Tool for Building Apps Using APIs
March 22, 2023 | Mike Vizard

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

GET THE TOP STORIES OF THE WEEK

Most Read on DevOps.com

Grafana Labs Acquires Pyroscope to Add Code Profiling Capability
March 17, 2023 | Mike Vizard
Four Technologies Transforming Data and Driving Change
March 17, 2023 | Thomas Kunnumpurath
How Database DevOps Fuels Digital Transformation
March 17, 2023 | Bill Doerrfeld
Neural Hashing: The Future of AI-Powered Search
March 17, 2023 | Bharat Guruprakash
5 Unusual Ways to Improve Code Quality
March 20, 2023 | Gilad David Maayan
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.