In my earlier blog, “From QA to Continuous Testing,” I explained that “several industry developments converged to make continuous testing the winning method for software engineering quality assurance (QA).” The “combination of improved test methods, agile processes, continuous integration speed afforded by automation and orchestration tools and low-cost test dynamically configurable infrastructures afforded by virtualization has resulted in DevOps.”
However, a recent survey by Parasoft indicated that some important concerns such as non-functional requirements (NFRs) (examples are: reliability, testability, availability, resiliency) are being overlooked in many organizations that are operating fast-paced Agile and DevOps environments.
Some results gleaned from Parasoft’s survey run counter to popular expectations:
1. Agile teams are less likely to monitor NFRs than non-agile teams.
2. Ownership of quality, especially for agile teams, tends to be QA team members.
3. Business members think NFRs are being measured but technical folks do not.
4. Confusion and disagreements exist regarding who owns NFR objectives.
The report implies that quick Agile and DevOps processing cycles steal time from broader QA concerns.
The Question
So what is the truth? Do agile, DevOps and continuous testing improve QA, or do they undermine quality?
My own observation is that DevOps can improve or undermine QA. The outcome depends on whether an organization explicitly takes a holistic view of quality that assures all of the “ilities” are taken care of during agile, DevOps and continuous testing.
A complete DevOps environment requires a more sophisticated view of continuous testing than simply “shift left,” “fail early,” “fail often” and “test fast,” which are four popular tenets espoused by the DevOps community. A fifth tenet, “relevance,” must also be fully considered. With “relevance” as a tenet of DevOps QA, the entire DevOps and continuous testing pipeline has woven into it, from end to end, the concept that all of the quality aspects relevant to customers are emphasized in continuous testing and analysis of test results. When relevance is fully considered as an equal tenet to the others, then NFRs and “ilities” are not forgotten and the tools, goals and tests explicitly take into account all aspects relevant to customers’ perception of quality. What is more, when relevance is applied to continuous test selection and results analysis, the system not only produces better quality but it also is more effective and efficient.
At Spirent we think that a holistic approach to DevOps and continuous testing, including “relevance,” is critical for successful DevOps. When continuous testing is implemented with relevance as an equal tenet, quality is improved.
You can read more about our views at Spirent.com/solutions/DevOps
What are your experiences and suggestions?
