Why Your Test Automation Is Failing

If you have been investing in test automation, but it’s not getting you where you think it should, it could be that you approached it superficially.

One of the most common issues is that the test part of test automation is rather straightforward to automate, especially with today’s abundance of great automation tools. But jumping straight into automated testing leaves you with a big heap of automation scripts, and often not the best ROI. Understanding why requires looking into the automation process.

When an engineer builds an automated test, they do it in a test environment that they (or someone in the organization) set up. Usually, this is the same way manual testers get access to test environments.

Before the engineers start automating, they have a working copy of the product they are testing, with relevant data on it. They have a good idea of the test they want to write. Everything looks promising.

They write the test and give it a few runs. Everything is looking nice and stable. Now, to provide the ROI we were hoping for, the test must run a million times. Or, at least, a few thousand times, right?

To reap the benefits of automation, it needs to happen frequently enough. However, the bottleneck is often discovered after creating a bunch of tests. Because the tests must run on a similar test environment to the one used to create them, with the new versions of the product or with any introduced changes that are being tested, you need access to hundreds of environments. This requirement makes it difficult to test effectively.

Below are three test environment approaches that inhibit test automation.

Manually Setting Up an Environment Each Time to Run Automated Tests

For tests that don’t need to run as frequently, such as security testing and performance testing, it’s common to set up an environment for automated tests every time the test needs to be executed. As a result, testers must wait for the environment to be ready, perhaps do some additional configurations and then run a suite of tests. And, if they’re trying to optimize utilization, the environment is left to run overnight or weekend.

The problem with this approach is it takes the “auto” out of automated testing. When automation depends on a manual step, the test is only as efficient as this manual step.

If the manual step isn’t very efficient, automation may not be helping at all. I’ve seen companies take so long to get an environment to work that running automated tests on it didn’t make any difference from a time perspective.

Setting Up a Static Environment That Automation Always Runs On

When setting up an environment takes a long time, it’s natural to assume it’s worth leaving an environment running so test automation can run on it. The preparations were made, the infrastructure is all in place, the product is configured and the data is ready, so let the automation party begin.

While complex, static test environments may seem ideal, but the party doesn’t last long. First, since it usually takes some time to put the environment together (same as the previous case), it often becomes high-maintenance. It can be scary to make changes that could ruin things, and you can end up needing someone to manage change full-time. Much like chocolate cake in the office kitchen, an environment that mimics production with good data on it tends to attract everyone. Soon, these nice environments become shared assets and tests start conflicting for the same resources. The natural next step is to set up another static environment. This loop usually ends with an angry manager saying something about cutting CAPEX.

Automating the Environment and Cutting Ourselves Some Slack

With the rise of infrastructure as code (IaC) and immutable infrastructure, there are more ways to spin up a test environment. Resourceful test teams will try to automate the environment creation or use built-in test tool or CI system capabilities to run tests on a public cloud instance or a container, if the product they test allows that. This is cheap and useful—and could work for some simple tests. The challenge usually starts with everything this approach doesn’t cover. It can be difficult to use IaC to spin up a test environment and handle all aspects of it—from security and isolation to dependencies, data, third-party integration and more. This is where this approach is often blocked and sends us back to the previous two approaches for a big part of the testing load—which again makes test automation investment less fruitful.

The Right Way to Set Up Test Environments: Test Automation on Dynamic Environments

The test automation world needs a game changer. You can make your car faster, but if the road is full of potholes, you won’t be able to get that fast or far. Test automation is similarly limited by the pace of environment creation, with its infrastructure, application, data and all.

This challenge becomes critical when adopting DevOps and testing stops being a stage in the pipeline, instead becoming an overall pipeline service. QA is no longer about being the gate between development to operations. It’s about overseeing the quality of the product during its entire life cycle. This means automated tests should always be able to run, and the environment they run on must be dynamic.

The solution to this challenge is environments as a service (EaaS). EaaS is all about packaging the environment, including the application, infrastructure and data, as an entity that can be provided as a service to a business process. Once defined, it can be provided on-demand for manual or automated users. Since it’s a well-defined entity, it also can be decommissioned when the process finishes using it. Playing an important role in making infrastructure horizontal, EaaS is also the key for paving the road for brilliant test automation.

If you think you should get more from your test automation initiative, it may be a good time to make EaaS part of your automation strategy.

— Maya Ber Lerner