DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust
  • 5 Key Performance Metrics to Track in 2023
  • Debunking Myths About Reliability
  • New Relic Bets on AI to Advance Observability
  • Vega Cloud Commits to Reducing Cloud Costs

Home » Features » The Software Testing Metrics That Make Or Break DevOps Success

The Software Testing Metrics That Make Or Break DevOps Success

By: Ericka Chickowski on July 31, 2018 1 Comment

Organizations today are racing to develop software at a breakneck pace through efficient DevOps processes. But it’s important for them to avoid losing sight of the forest for the trees. If too much of that speed comes with quality sacrifices that degrade the user experience, the rapid software iterations hurt the business more than they help it.

Recent Posts By Ericka Chickowski
  • 5 Ways DevSecOps Can Manage Software Supply Chains
  • 4 Traits of High-Performance Digital Leaders
  • Are Self-Service Machine Learning Models the Future of AI Integration?
More from Ericka Chickowski
Related Posts
  • The Software Testing Metrics That Make Or Break DevOps Success
  • The Role of QA in the DevOps World
  • Focusing on the True DevOps North Star: Continuous Improvement
    Related Categories
  • DevOps Practice
  • Features
    Related Topics
  • devops
  • forrester
  • metrics
  • testing
Show more
Show less

“Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” said Clark Golestani, former CIO at pharmaceuticals giant Merck and current managing director of C Sensei Group. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors.”

TechStrong Con 2023Sponsorships Available

This is why continuous and automated testing is so essential to DevOps success. But instituting effective and streamlined testing is easier said than done. Organizations must continually tweak what, how and when they test their software to achieve the optimal test coverage while introducing as little friction as possible into the software delivery pipeline. It’s a delicate balance, and the only way to achieve it is to keep meticulous track of metrics that evaluate how well the testing process is improving quality throughout the development life cycle.

“As the saying goes, ‘You can’t manage what you don’t measure,'” said Andrew Magarie, senior market impact consultant at Forrester Consulting. “So firms need to be able to define and track the right metrics that can help enable quality at speed while minimizing risk.”

Magarie and his team at Forrester recently conducted research into 94 different quality metrics frequently measured by DevOps organizations across four development stages: build, functional validation, integration and end-to-end regression testing. The study showed that the most effective test metrics that bubbled up across all of stages tended to clump into at least one of three categories.

Risk-Focused Metrics

According to Magarie’s report, the most mature and effective organizations fundamentally chose testing metrics that best helped them understand business risk posed by the software throughout the development process.

“For this study, we use the following definition of business risk: Any application shortcoming that impairs the end user’s (or customer’s) expected experience and ultimately erodes confidence in the business,” he explained.

For example, during the build stage mature organizations consider measuring the number of automated tests prioritized by risk as one of the most important assessments.

“Tracking ‘unit’ tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk,” wrote Magarie.

As he explained, business risk will be different for every application, and the complexity of an organization’s architecture and transaction dependencies adds more variables to the equation. However, organizations need to seek ways to measure ‘relative risk’ across the software development life cycle (SDLC) if they truly want to take a disciplined approach to incrementally improving quality in their software.

Contextual Metrics

Forrester found that simple “counting” metrics don’t offer QA nearly as much value as contextual metrics. So simply tracking the number of tests will not provide teams with as much actionable data as the measurement of the ratio of requirements covered by tests. Approximately 69 percent of mature organizations say running functional tests and checking against covered requirements is a top measurement and 75 percent say the same about API tests run against functional requirements.

Holistic UX Metrics

The most mature organizations recognize that while application-specific or team-specific QA metrics are important, the ultimate goal is ensure the quality of the user experience (UX) across their entire interaction with an application. This means measuring the thoroughness of end-to-end regression testing is crucial. Approximately 70 percent of leading DevOps organizations say they believe the percent of automated end-to-end test cases as a top metric.

— Ericka Chickowski

Filed Under: DevOps Practice, Features Tagged With: devops, forrester, metrics, testing

« Postman Announces Team Plan for Free Users
How to Use the Jenkins Pipelines »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

https://webinars.devops.com/overcoming-business-challenges-with-automation-of-sap-processes
Tuesday, April 4, 2023 - 11:00 am EDT
Key Strategies for a Secure and Productive Hybrid Workforce
Tuesday, April 4, 2023 - 1:00 pm EDT
Using Value Stream Automation Patterns and Analytics to Accelerate DevOps
Thursday, April 6, 2023 - 1:00 pm EDT

Sponsored Content

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Practical Approaches to Long-Term Cloud-Native Security

December 5, 2019 | Chris Tozzi

Latest from DevOps.com

npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust
March 31, 2023 | Richi Jennings
5 Key Performance Metrics to Track in 2023
March 31, 2023 | Sarah Guthals
Debunking Myths About Reliability
March 31, 2023 | Kit Merker
New Relic Bets on AI to Advance Observability
March 30, 2023 | Mike Vizard
Vega Cloud Commits to Reducing Cloud Costs
March 30, 2023 | Mike Vizard

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

GET THE TOP STORIES OF THE WEEK

Most Read on DevOps.com

Don’t Make Big Tech’s Mistakes: Build Leaner IT Teams Instead
March 27, 2023 | Olivier Maes
How to Supercharge Your Engineering Teams
March 27, 2023 | Sean Knapp
Five Great DevOps Job Opportunities
March 27, 2023 | Mike Vizard
The Power of Observability: Performance and Reliability
March 29, 2023 | Javier Antich
Cloud Management Issues Are Coming to a Head
March 29, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.