A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

State of DevOps Maturity

Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

Fully Automated Testing

Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

Key Takeaways

DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.