DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv Video Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Container Journal
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv Video Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • Continuous Delivery
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Platform Engineering: Creating a Paved Path to Reduce Developer Toil
  • Running Serverless in Production: 7 Critical Best Practices
  • Where Does Observability Stand Today, and Where is it Going Next?
  • Five Great DevOps Job Opportunities
  • A Freelancer's Workflow

Home » Blogs » Continuous Testing – Accelerated !

Continuous Testing – Accelerated !

By: Marc Hornbeek on March 10, 2015 1 Comment

Except for cases where test duration is the point of the test, as indicated in my prior blog “Isn’t Continuous Testing a Longevity Test Anti-pattern?”, faster continuous testing cycles are preferred because failing early and often is a DevOps goal. There are multiple reasons for this, but it should be evident that faster tests reduce the wait time to get results and since there are typically many development team members looking for results the developer time savings multiplier effect is huge for every minute of test time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test can be man-years of time saved over a year!

Recent Posts By Marc Hornbeek
  • Optimizing Cloud Costs for DevOps With AI-Assisted Orchestration
  • Best of 2021 – Nine Pillars of DevOps Best Practices
  • Zero-Trust Network Access Platforms Slash DevOps Bottlenecks
More from Marc Hornbeek
Related Posts
  • Continuous Testing – Accelerated !
  • Attaining Continuous Testing in DevOps through a Templated Approach
  • The Surprising Secret to DevOps ROI
    Related Categories
  • Blogs
  • Continuous Testing
  • DevOps Toolbox
    Related Topics
  • automation
  • continuous delivery
  • continuous integration
  • continuous testing
  • devops
  • QA
  • testing
  • tools
Show more
Show less

So what can be done to accelerate continuous testing? Frankly a thorough answer to this topic could fill a book. Below are some suggestions in a checklist format intended to be useful for DevOps .

TechStrong Con 2023Sponsorships Available
  1. Organize the team for speed: When team members and the organization are explicitly selected for CT speed the entire DevOps process will be accelerated.
    • A test architect familiar with the principles of best practices test environments and continuous test design is critical to design and tune continuous testing for speed. The architect establishes the CT strategy and speed goal relative to the coverage goal.
    • A test analysis team reviews test results so no time is wasted on results analysis.
    • Developers and test members are trained on best practices test design and their test scripts are peer-reviewed to ensure they comply with the strategy.
  2. Select powerful test resources: Powerful, dedicated, scalable resources will accelerate all test processes and shorten the time to results.
    • Select the fastest processors, low latency storage and I/O to initialize and execute both the test systems and system-under-test entities quickly.
    • Dedicate resources that have longer initialization times so they can be pre-initialized.
    • Virtualize resources that can be scaled horizontally without sacrificing speed.
    • If affordable, arrange enough resources to run ALL tests in parallel.

  3. Choose test tools that are designed for fast test execution: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
    • Restful APIs that do not require wait states between commands and responses
    • Priority queuing of test cases
    • Creation of test topologies that can be pre-loaded
    • Ability to spawn out tests in parallel to run on any number of machines
    • The ability to park a test that needs to wait for an event and return to it after proceeding with subsequent tests
    • Caching features that enable initialization and results data to be processed for one test while the tool proceeds to execute other tests
    • Multiple devices, session types and results data processing utilities implemented as an embedded part of the tool and not requiring slow speed scripting
    • The ability to render commonly used test utility functions in a fast execution language.

  4. Organize test schedules and test suites to fail early: The sequence and approach for scheduling of test cases greatly affects the speed of testing. If possible the scheduler should launch all tests in parallel as soon as the build is ready. Otherwise priority schemes are needed.
    • Organize the test scheduler and test suites to avoid any unnecessary tests.
    • Cache initialization and topologies data and resources ahead of the next test execution.
    • Check the test environment before running tests, abort if invalid conditions are detected.
    • Organize tests so the most important ones are run first.
    • Set a failure escape threshold and abort the test if exceeded.
    • Eliminate dependencies between tests so that each one can run standalone or in parallel. Queue the next test while a test is running.
    • Use smart gray box testing methods to limit test selection to the most important tests according to the software change and skip tests that are low risk.
    • Run longer tests in parallel with shorter ones.

5.Design test cases to fail early: The test cases themselves need to be designed with the strategy to get test results as fast as possible.
• Design each test case with a single unique test purpose and make it self-contained so it may be scheduled to run standalone or in parallel with any other test case.
• If a test case has multiple verdicts arrange the highest priority verdicts first and close the test at the first failure.
• Prune or refactors tests that have a low failure history.
• Shorten dead-script timers, and make any wait timers conditional so they can expire as soon as the condition is met.
• Reduce the number of test iterations to fit a time budget.
• Reduce communication wait times by using the fastest protocols. Optimize the most frequently used utilities.

  1. Arrange test results reduce to minimize analysis wait times: Arrange the test results to be consumable as soon as the tests are run so the system does not bottleneck.
    • Report test results as the tests are run and accumulate the results across parallel test targets.
    • Establish thresholds for aggregated test results and abort the tests and release test resources to be used by other tests if the thresholds are exceeded for any run.
    • Keep track of test timings and adjust the test schedules according the average and peak run times for each test case.
    • Separate test result report processes to persistent resources so no test process is waiting to finish a report and the test process can proceed to the next test in parallel with test results.
    • Correlate the test results to the changes and use trends analysis to help speed up diagnostics and as an input to test selection such that the most failing test cases are given higher weight in future test runs.

The above is a partial list of suggestions for accelerating continuous testing that have been proven to yield good results for accelerating continuous testing and DevOps.

At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

What do you think of these suggestions and do you have others that should be mentioned?

Filed Under: Blogs, Continuous Testing, DevOps Toolbox Tagged With: automation, continuous delivery, continuous integration, continuous testing, devops, QA, testing, tools

« Red Hat launches platform to support Docker, Kubernetes and enterprise-grade containerization
First wave of sessions and speakers announced for DevOps Connect: SecDevOps @ RSA Conference »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Automating Day 2 Operations: Best Practices and Outcomes
Tuesday, February 7, 2023 - 3:00 pm EST
Shipping Applications Faster With Kubernetes: Myth or Reality?
Wednesday, February 8, 2023 - 1:00 pm EST
Why Current Approaches To "Shift-Left" Are A DevOps Antipattern
Thursday, February 9, 2023 - 1:00 pm EST

Sponsored Content

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Why a Low-Code Platform Should Have Pro-Code Capabilities

March 24, 2021 | Andrew Manby

AWS Well-Architected Framework Elevates Agility

December 17, 2020 | JT Giri

Practical Approaches to Long-Term Cloud-Native Security

December 5, 2019 | Chris Tozzi

Latest from DevOps.com

Where Does Observability Stand Today, and Where is it Going Next?
February 6, 2023 | Tomer Levy
Five Great DevOps Job Opportunities
February 6, 2023 | Mike Vizard
Azure Migration Strategy: Tools, Costs and Best Practices
February 3, 2023 | Gilad David Maayan
Blameless Integrates Incident Management Platform With Opsgenie
February 3, 2023 | Mike Vizard
OpenAI Hires 1,000 Low Wage Coders to Retrain Copilot | Netflix Blocks Password Sharing
February 2, 2023 | Richi Jennings

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

GET THE TOP STORIES OF THE WEEK

Most Read on DevOps.com

OpenAI Hires 1,000 Low Wage Coders to Retrain Copilot | Netflix Blocks Password Sharing
February 2, 2023 | Richi Jennings
Automation Challenges Holding DevOps Back
February 1, 2023 | Mike Vizard
New Relic Bolsters Observability Platform
January 30, 2023 | Mike Vizard
Jellyfish Adds Tool to Visualize Software Development Workflows
January 31, 2023 | Mike Vizard
Cisco AppDynamics Survey Surfaces DevSecOps Challenges
January 31, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.