Get Testing Bottlenecks Out of Your Pipelines

I first learned the term “deployment pipeline” when I read “Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation,” by Jez Humble and David Farley back in 2009. End-to-end automation of the build, deploy, test and release processes kick in each time a team member commits a change to the source code repository.

Chapter 5 of the book notes that the forms of testing common to all projects include manual test stages which “… assert that the system is usable and fulfills its requirements, detect any defects not caught by automated tests, and verify that it provides value to its users. These stages might typically include exploratory testing environments, integration environments, and UAT (user acceptance testing).”

Humble and Farley include these manual stages in the deployment pipeline, also known as a continuous integration (CI) pipeline, build pipeline or delivery pipeline. However, from what I observe, I think a lot of times people forget about these important manual stages.

Though we can do more and more with test automation, thanks to improving technology, manual testing activities are here to stay in most business domains. My own previous team realized this as we started to move toward continuous delivery (CD) a couple of years back, with twice-weekly deploys to production. Smaller changes and more frequent deploys means lower risk of regression failures. Yet, we still struggled with how to fit in important manual testing activities.

Whenever my cross-functional delivery team has a problem, I look for a way to make it visible, to help us talk about it and brainstorm ideas to solve it. My “aha” moment around manual stages came when I learned techniques to visualize our pipeline from Abby Bangser and Ashley Hunsberger. My team knew we had these manual testing activities and that we wouldn’t feel confident about releasing changes without them, but we weren’t really thinking about them as stages in the pipeline.

Visualizing Your Pipeline – Including the Manual Stages

Getting your team together to visualize your pipeline is a great method to find ways to improve it. Gather around a table with cards or sticky notes, or draw on a whiteboard—what are all the steps in your pipeline, including the manual ones? In Figure 1, the manual stages in a team’s CD pipeline are shown in yellow. All the automated and manual things need to happen before the business decides to push the button and deploy to production. It’s easy to see that these manual stages can become a bottleneck. Manual exploratory testing is likely to take longer than the automated test suites. If user acceptance testing needs to be done by certain business stakeholders and they aren’t available at the right time, that holds up the release. If the team is looking to move to continuous deployment, where every change to the product goes out to production as soon as all stages complete successfully, these manual stages will be in the way.

When you get your whole team including developers, database experts, operations specialists and product owners involved in solving these problems, you may discover innovative ways to approach them. In my team’s case, we decided everyone on the team needed to help with exploratory testing so we could do it in a timely manner. Testers helped the developers learn exploratory testing skills with hands-on workshops and by pairing with them. Then, each developer pair working on a story was tasked with doing the exploratory testing at the story level before declaring the story finished. We wrote exploratory testing charters at the epic or feature level as stories in our backlog. When enough stories were complete, anyone on the team, including testers and product owners, could pick up those charters.

If we couldn’t complete all the necessary exploratory testing before the next production deploy, we used feature flags to hide the changes from customers until we could complete the testing. We could go ahead and deploy the changes and wait to release them once we felt confident they would not cause any customer pain.

Asynchronous Testing Stages

Using feature flags or toggles is a good way to allow the team to approach some manual testing activities asynchronously, so they don’t hold up the deployment pipeline. If some required manual accessibility, testing takes an extra day to complete or the UAT has to be postponed for a week, the deploy still happens. A change could be toggled on for a small percentage of users and observed in production to see if any unexpected behavior occurs. Whatever activities are needed to safely release a particular change don’t have to hold up other changes that are ready for customer use.

Figure 2 is an example of a continuous deployment pipeline, but this asynchronous approach to the manual stages works for continuous delivery as well. Each time a team member commits a change, if all the automated stages in the pipeline pass, the release candidate build will be deployed to production. The manual testing activities are not inline with the other steps and don’t hold back the deployment.

If we’re ready to deploy a feature but need to finish those manual testing stages before we release it, developers can put in a feature flag set to “off” to prevent it from being exposed to customers. When the asynchronous manual stages are all complete and successful, the feature flag can be flipped on in production. This is the actual release, customers can now use the feature. The feature flag should be removed as soon as possible, as too many feature flags at once has serious risks.

Using feature flags or toggles to permit deploying changes to production before all testing stages are finished is one way to avoid bottlenecks and succeed with continuous delivery or deployment. Continuous delivery and deployment success depend on doing frequent small changes, not big-bang releases. A lean approach to feature flags, keeping the inventory low, is advisable.

Visualize It With Your Team

We’ve looked at one way to keep manual pipeline stages from holding up deploys, but there may be other solutions for your team. And there are lots of other key success factors for CD, such as keeping changes small and frequent. There’s lots to discuss and brainstorm about as your team, or a representational cross-section of it, gets together and experiments with a visual model of your pipeline. Talk about what you need to learn from each pipeline stage, who needs to know it and whether the pipeline is providing actionable feedback.

As a tester, I’ve sometimes questioned aspects of the pipeline that nobody else had thought about. Having team members with development, operations, database and other skills present means more potential for creative solutions. Our pipeline provides a major feedback loop and we want that feedback as fast as we can get it. Writing each stage on a card and arranging them on a table or wall often triggers ideas for parallelizing more stages or realizing a particular stage isn’t necessary or can be done locally by developers.

Automate all the things where it’s feasible. Use the time you save to optimize the way you accomplish manual testing activities and possibly automate more of them. Master techniques such as feature flagging so that you can take the time needed to feel confident about every new feature you release.

References

“Continuous Delivery: Reliable Software Releases through Build, Test and Deployment Automation,” Jez Humble and David Farley, https://continuousdelivery.com

“A Practical Guide to Testing in DevOps,” Katrina Clokie, https://leanpub.com/testingindevops

“Feature Toggles, AKA Feature Flags,” Pete Hodsgon, https://martinfowler.com/articles/feature-toggles.html

— Lisa Crispin