DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • Calendar View
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • Calendar View
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Cloud Native Now
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • CI/CD
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Sustainability
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Atlassian Advances DevSecOps via Jira Integrations
  • PagerDuty Signals Commitment to Adding Generative AI Capabilities
  • Mastering DevOps Automation for Modern Software Delivery
  • DigiCert Allies With ReversingLabs to Secure Software Supply Chains
  • The Future of Continuous Testing in CI/CD

Home » Blogs » DevOps Toolbox » Continuous test monitoring DevOps ”health-beat”

Continuous test monitoring DevOps ”health-beat”

Marc Hornbeek ProfileBy: Marc Hornbeek on September 11, 2015 3 Comments

In my prior blog Test Results Analysis at the Speed of DevOps I discussed ways that test analysis features and practices can be optimized to ensure that test results for each CI/CT cycle keep pace with accelerated continuous testing. Beyond optimizing continuous test analysis for each cycle, longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much. Some of the more difficult problems to detect and diagnose are those that only occur occasionally or under “corner case conditions”.

Recent Posts By Marc Hornbeek
  • Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
  • How Test-Driven Methodologies Reduce CI/CD Lead Time
  • AI Chatbots: Game Changers for DevOps Transformations
Marc Hornbeek Profile More from Marc Hornbeek
Related Posts
  • Continuous test monitoring DevOps ”health-beat”
  • Test Results Analysis at the Speed of DevOps
  • Continuous Testing System Stability
    Related Categories
  • Blogs
  • DevOps Toolbox
    Related Topics
  • continuous testing
  • devops tools
  • monitoring
  • tools
Show more
Show less

Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles then there is no way to build sufficient sustainable confidence in the long term health of the DevOps pipeline or products produced by the pipeline. The combination of continuous test monitoring tools with continuous testing and analysis tools can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in the product for each release and its evolution over multiple releases. Below are some suggestions in a checklist format that have proven useful for continuous test monitoring for sustainable high performance DevOps environments.

Cloud Native NowSponsorships Available
  1. Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies, and performance degradation within the margin of a test but accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.
  2. Regression test product areas even though there were no expected changes: One of the practices in high performance DevOps environments is to skip tests for areas that are “known” not to have changed during individual CI/CT cycles. Unintended consequences of indirect changes may impact performance so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility features and upgrade/downgrade administration features.

  3. Select continuous test monitoring tools that collect and report trends: Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases, and functional tags can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.

  4. Use continuous test monitoring results to diagnose and resolve problems: Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.

The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable DevOps. At Spirent we think testing has a bright future in DevOps. You can hear more about our views on this topic by reviewing our joint webinar conducted together with test management product provider Zephyr at: Overcoming-DevOps Challenges

What do you think of these suggestions and do you have others that should be mentioned?

Filed Under: Blogs, DevOps Toolbox Tagged With: continuous testing, devops tools, monitoring, tools

« The Third “S” To A Successful App: Street Smarts
How do large enterprises do DevOps? Recap of my third video chat with the 2015 DevOps Enterprise Summit Speakers »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Maximize IT Operations Observability with IBM i Within Splunk
Wednesday, June 7, 2023 - 1:00 pm EDT
Secure Your Container Workloads in Build-Time with Snyk and AWS
Wednesday, June 7, 2023 - 3:00 pm EDT
ActiveState Workshop: Building Secure and Reproducible Open Source Runtimes
Thursday, June 8, 2023 - 1:00 pm EDT

GET THE TOP STORIES OF THE WEEK

Sponsored Content

PlatformCon 2023: This Year’s Hottest Platform Engineering Event

May 30, 2023 | Karolina Junčytė

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Latest from DevOps.com

Atlassian Advances DevSecOps via Jira Integrations
June 6, 2023 | Mike Vizard
PagerDuty Signals Commitment to Adding Generative AI Capabilities
June 6, 2023 | Mike Vizard
Mastering DevOps Automation for Modern Software Delivery
June 6, 2023 | Krishna R.
DigiCert Allies With ReversingLabs to Secure Software Supply Chains
June 6, 2023 | Mike Vizard
The Future of Continuous Testing in CI/CD
June 6, 2023 | Alexander Tarasov

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Most Read on DevOps.com

No, Dev Jobs Aren’t Dead: AI Means ‘Everyone’s a Programmer’? ¦ Interesting Intel VPUs
June 1, 2023 | Richi Jennings
Forget Change, Embrace Stability
May 31, 2023 | Don Macvittie
Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools
June 2, 2023 | Marc Hornbeek
Friend or Foe? ChatGPT’s Impact on Open Source Software
June 2, 2023 | Javier Perez
Checkmarx Brings Generative AI to SAST and IaC Security Tools
May 31, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.