DevOps Places Software Testing in the Crosshairs

To keep up with the pace of innovation, testing must play a larger role in the software development life cycle

The speed of digital transformation is already staggering, and it’s only going to increase. To put this into some very concrete terms, consider that:

• There are 7.7 billion people in the world.
• 5 billion have regular access to a toilet.
• 5 billion own a mobile phone.

All of a sudden, a huge number of people jumped from a very provincial lifestyle straight into digital times—creating a tremendous demand for more, and more innovative, software.

The traditional ways of developing and delivering software aren’t adequate for meeting this new demand. Not long ago, most companies were releasing software annually or bi-annually. Now, iterations commonly last two weeks or less. While delivery cycle time is decreasing, the technical complexity required to deliver a positive user experience and maintain a competitive edge is increasing.

In terms of software testing, this brings us to an inflection point. In most organizations, testers were already racing to keep pace when delivery cycles were longer and application complexity was lower. Every quarter or so, the development team would pass a release candidate over the wall to QA, who would then scramble to validate it as thoroughly as possible in the allotted time—largely with manual testing.

Now, digital transformation initiatives such as Agile and DevOps are pushing traditional methods to their breaking point. Organizations are releasing much more frequently—from monthly on the low end to multiple times per hour on the high end. And as organizations increasingly edge toward continuous delivery with automated delivery pipelines, intermediary quality gates and the ultimate go/no-go decisions will all hinge upon test results.

Is your organization’s testing process up to the task?

We Have a Testing Problem (Two, Actually)

In most organizations, testing delays application delivery while providing limited insight into whether the applications are meeting stakeholder expectations. It’s not fast enough to help teams find and fix defects when it’s optimal to do so. And it’s reporting on low-level failures (e.g., 78% of our tests passed) rather than providing the business-focused perspective needed to make fast release decisions (e.g., Only 38% of our business risk was tested … and 25% of that didn’t work properly).

Let’s take a quick look at each of these problems in turn.

The Speed Problem

DevOps is all about removing the barriers to delivering innovative software faster. Yet, while other aspects of the delivery process are streamlined and accelerated, testing consistently emerges as the greatest limiting factor.

A recent GitLab survey that targeted developers and engineers found that testing is responsible for more delays than any other part of the development process.

Where in the development process do you encounter the most delays?

The same conclusion was reached by a DevOps Review survey that polled a broader set of IT leaders across organizations practicing DevOps. Again, testing was cited as the No. 1 source of holdups in the software delivery process. In fact, it “won” by a rather wide margin here—63% reported that testing was a major source of delays, while the second-highest source of delays (planning) was cited by only 32% of the respondents.

Where are the main holdups in the software production process?

Why is testing such a formidable bottleneck? That could be the topic of an entire book. For now, let’s summarize some key points:

• The vast majority of testing (over 80%) is still performed manually—even more at large enterprise organizations.
• Approximately 67% of the test cases being built, maintained and executed are redundant and add no value to the effort.
• At organizations that have significant test automation, testers spend 17% of their time dealing with false positives and another 14% on additional maintenance tasks.
• Over half of testers spend five to 15 hours per week dealing with test data (average wait time for data = 2 weeks).
• 84% are routinely delayed by limited test environment access (average wait time for test environments = 32 days).
• The average regression test suite takes 16.5 days to execute, but the average Agile sprint is two weeks, from start to finish—including planning, implementation and testing.
• The average application under test now interacts with 52 dependent systems—which means that a single end-to-end transaction could cross everything from microservices and APIs to a variety of mobile and browser interfaces, packaged apps (SAP, Salesforce, Oracle, ServiceNow…), custom/legacy applications and mainframes.

The software testing process wasn’t working perfectly even before the advent of Agile and DevOps. Now we’re asking teams to “just speed it up” at the same time that modern application architectures are making the process even more complex. Given that, it’s hardly surprising that speed expectations aren’t being met.

The Insight Problem

Only 9% of companies perform formal risk assessments on their requirements/user stories. Most attempt to cover their top risks intuitively, and this results in an average business risk coverage of 40%. Would you feel comfortable driving a race car with blinders on? That’s essentially what you’re doing if you’re rapidly delivering software with insight into less than half of your total business risk.

Moreover, most organizations can’t immediately differentiate between a test failure for a trivial issue and a business-critical failure that must be addressed immediately. Most results look something like this:

What insight does that really provide? It’s clear that …

• There’s a total of 53,274 test cases.
• Almost 80% (42,278) passed.
• Over 19% failed.
• About 1% did not execute.

Maybe the test failures are related to some trivial functionality. Maybe they stem from the most critical functionality: the “engine” of your system. Or, maybe the most critical functionality was not even tested at all. Trying to track down this information would require tons of manual investigative work that yields delayed, often-inaccurate answers.

Today’s go/no-go decisions need to be made rapidly—even automatically and instantaneously. Results that focus on the number of test cases leave you with a huge blind spot that becomes absolutely critical—and incredibly dangerous—when you’re moving at the speed of digital business.

Closing the Gap

How do you evolve from the slow, burdensome testing that delivers questionable results to the lean, streamlined testing that provides the fast feedback required to accelerate innovation and delivery?  That’s what I aim to explain in my new book, “Enterprise Continuous Testing: Transforming Testing for Agile and DevOps.” This book is targeted to senior quality managers and business executives who need to achieve the optimal balance between speed and quality when delivering the software that drives the modern business. It provides a road map for how to accelerate delivery with high confidence and low business risk.

— Wolfgang Platz