Except for cases where test duration is the point of the test, as indicated in my prior blog “Isn’t Continuous Testing a Longevity Test Anti-pattern?”, faster continuous testing cycles are preferred because failing early and often is a DevOps goal. There are multiple reasons for this, but it should be evident that faster tests reduce the wait time to get results and since there are typically many development team members looking for results the developer time savings multiplier effect is huge for every minute of test time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test can be man-years of time saved over a year!
So what can be done to accelerate continuous testing? Frankly a thorough answer to this topic could fill a book. Below are some suggestions in a checklist format intended to be useful for DevOps .
- Organize the team for speed: When team members and the organization are explicitly selected for CT speed the entire DevOps process will be accelerated.
• A test architect familiar with the principles of best practices test environments and continuous test design is critical to design and tune continuous testing for speed. The architect establishes the CT strategy and speed goal relative to the coverage goal.
• A test analysis team reviews test results so no time is wasted on results analysis.
• Developers and test members are trained on best practices test design and their test scripts are peer-reviewed to ensure they comply with the strategy. Select powerful test resources: Powerful, dedicated, scalable resources will accelerate all test processes and shorten the time to results.
• Select the fastest processors, low latency storage and I/O to initialize and execute both the test systems and system-under-test entities quickly.
• Dedicate resources that have longer initialization times so they can be pre-initialized.
• Virtualize resources that can be scaled horizontally without sacrificing speed.
• If affordable, arrange enough resources to run ALL tests in parallel.Choose test tools that are designed for fast test execution: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
• Restful APIs that do not require wait states between commands and responses
• Priority queuing of test cases
• Creation of test topologies that can be pre-loaded
• Ability to spawn out tests in parallel to run on any number of machines
• The ability to park a test that needs to wait for an event and return to it after proceeding with subsequent tests
• Caching features that enable initialization and results data to be processed for one test while the tool proceeds to execute other tests
• Multiple devices, session types and results data processing utilities implemented as an embedded part of the tool and not requiring slow speed scripting
• The ability to render commonly used test utility functions in a fast execution language.Organize test schedules and test suites to fail early: The sequence and approach for scheduling of test cases greatly affects the speed of testing. If possible the scheduler should launch all tests in parallel as soon as the build is ready. Otherwise priority schemes are needed.
• Organize the test scheduler and test suites to avoid any unnecessary tests.
• Cache initialization and topologies data and resources ahead of the next test execution.
• Check the test environment before running tests, abort if invalid conditions are detected.
• Organize tests so the most important ones are run first.
• Set a failure escape threshold and abort the test if exceeded.
• Eliminate dependencies between tests so that each one can run standalone or in parallel. Queue the next test while a test is running.
• Use smart gray box testing methods to limit test selection to the most important tests according to the software change and skip tests that are low risk.
• Run longer tests in parallel with shorter ones.
5.Design test cases to fail early: The test cases themselves need to be designed with the strategy to get test results as fast as possible.
• Design each test case with a single unique test purpose and make it self-contained so it may be scheduled to run standalone or in parallel with any other test case.
• If a test case has multiple verdicts arrange the highest priority verdicts first and close the test at the first failure.
• Prune or refactors tests that have a low failure history.
• Shorten dead-script timers, and make any wait timers conditional so they can expire as soon as the condition is met.
• Reduce the number of test iterations to fit a time budget.
• Reduce communication wait times by using the fastest protocols. Optimize the most frequently used utilities.
- Arrange test results reduce to minimize analysis wait times: Arrange the test results to be consumable as soon as the tests are run so the system does not bottleneck.
• Report test results as the tests are run and accumulate the results across parallel test targets.
• Establish thresholds for aggregated test results and abort the tests and release test resources to be used by other tests if the thresholds are exceeded for any run.
• Keep track of test timings and adjust the test schedules according the average and peak run times for each test case.
• Separate test result report processes to persistent resources so no test process is waiting to finish a report and the test process can proceed to the next test in parallel with test results.
• Correlate the test results to the changes and use trends analysis to help speed up diagnostics and as an input to test selection such that the most failing test cases are given higher weight in future test runs.
The above is a partial list of suggestions for accelerating continuous testing that have been proven to yield good results for accelerating continuous testing and DevOps.
At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops
What do you think of these suggestions and do you have others that should be mentioned?
