In my prior blog Test Results Analysis at the Speed of DevOps I discussed ways that test analysis features and practices can be optimized to ensure that test results for each CI/CT cycle keep pace with accelerated continuous testing. Beyond optimizing continuous test analysis for each cycle, longer term continuous test monitoring is necessary to assure overall delivery confidence for each release and longer term product performance trend in a positive direction. Individual test results or a fixed number of test campaigns can only tell you so much. Some of the more difficult problems to detect and diagnose are those that only occur occasionally or under “corner case conditions”.
Unless the results of testing and test analysis are continuously monitored and results aggregated over multiple test and release cycles then there is no way to build sufficient sustainable confidence in the long term health of the DevOps pipeline or products produced by the pipeline. The combination of continuous test monitoring tools with continuous testing and analysis tools can provide a longer term strategic view of test results that is necessary to collect, aggregate and organize test results data to gain confidence in the product for each release and its evolution over multiple releases. Below are some suggestions in a checklist format that have proven useful for continuous test monitoring for sustainable high performance DevOps environments.
- Determine continuous test monitoring priorities: Some examples of problems that continuous test monitoring can help with include intermittent failures caused by marginal designs, marginal test designs, environmental condition changes not detected by individual tests, memory leaks, varying starting conditions, interactions with other systems, system topologies, and performance degradation within the margin of a test but accumulate over time. The best practice for continuous monitoring indicates that the problems of most concern to a specific product or DevOps environment will be monitored.
Regression test product areas even though there were no expected changes: One of the practices in high performance DevOps environments is to skip tests for areas that are “known” not to have changed during individual CI/CT cycles. Unintended consequences of indirect changes may impact performance so a regression suite should audit these areas occasionally just to be sure. Typical examples that are often caught by this are features important audit functions, system backwards compatibility features and upgrade/downgrade administration features.
Select continuous test monitoring tools that collect and report trends: Tools that can correlate and report test results across multiple dimensions such as test cases, product features, system performance categories, build versions, releases, and functional tags can find intermittent bugs or problem trends. Thresholds and email alerts and dashboards that highlight short term results views from long term results views are especially valuable.
Use continuous test monitoring results to diagnose and resolve problems: Intermittent failures and problems that only become apparent when test results are presented as trends over a long term trend are easier to diagnose when a large data set is accumulated and filtered by most probable cause tags. Once a diagnosis is determined, the root cause can be verified through targeted retest cases that set all the conditions in accordance with the diagnosis. Once confirmed then the offending design can be refactored to handle or avoid the failure condition.
The above is a partial list of suggestions for continuous test monitoring that have been proven to support sustainable DevOps. At Spirent we think testing has a bright future in DevOps. You can hear more about our views on this topic by reviewing our joint webinar conducted together with test management product provider Zephyr at: Overcoming-DevOps Challenges
What do you think of these suggestions and do you have others that should be mentioned?
