In today’s economy, the companies that prevail are those that can accelerate software delivery and increase developer productivity. Yet, as applications become more complex and codebases grow, companies are forced to implement more comprehensive quality safeguards, which slow the rate at which they can deliver new software. Delivering more incremental builds in shorter intervals can address this conundrum: Customer needs are serviced quicker, overhead for engineers is reduced, and risk is mitigated because changes released into production are smaller.
That’s why high-velocity engineering teams are applying not only continuous delivery processes but also lessons in experimentation from established leaders including Amazon, Netflix and Facebook. These companies have made experimentation a foundation for their release processes, trying out major feature releases and redesigns within smaller groups before making them broadly available. By using new techniques such as feature flagging, rollouts and traffic splitting, experimentation is no longer just the future for marketing teams; it’s quickly becoming an essential practice for high-performing development teams as well.
Today, discussions around velocity focus on key practices that developers can use to maximize the effectiveness of continuous delivery, software development and experimentation programs. Here are some key ideas to consider.
Reducing Your Dependency on a QA Team Actually Makes You Faster (Shifting Left)
Many organizations eschew the conventional wisdom and deprioritize the heavy lifting of writing comprehensive test suites, opting instead to throw them over the fence to a Quality Assurance team. This approach is almost always less effective and causes organizations to lose momentum instead of conserving it.
Higher-performing teams actually “shift left” and empower developers to own many parts of the QA process. A solid foundation of tests creates less overhead later on and empowers engineers to release quality code into production faster. Companies can address customer and product needs efficiently, quickly and with more confidence than if they wait for full QA cycles to finish.
But what if you don’t have the ability (or luxury) to rewrite tens of thousands of tests? The good news is that even without expansive test coverage, you can still take advantage of this approach using code coverage tools. These work by evaluating how much of your codebase gets executed during test runs. If the overall coverage decreases, it means a feature was potentially rushed and is untested and should not be merged into the codebase until the ratio is balanced out. This code-to-test measurement can even “block” code merges unless it meets or exceed the previous threshold.
Prioritize the Build and Test Frameworks Themselves
Teams that that prioritize the development of build-and-test frameworks—as opposed to individual tests—are rewarded with faster, more reliable builds. But it’s a struggle to build out reliable processes that can handle tens of thousands of tests in a performant way for each developer.
Particularly with more complicated test suites, there’s a tendency for these tests to become brittle or prone to occasional failure because of all the moving pieces. Optimizely built a “test gatekeeper” that automatically disables tests that fail in this way, without the need for human intervention, and assigns them to an owner to be fixed. Remember: unreliable tests are sometimes worse than no test at all.
Achieve Faster Release Cycles Through Experimentation
Companies that want additional, real-time insights and fine-tuning for their application stack are shifting to feature flagging and feature rollouts to expand their capabilities.
Feature flags are simple control-flow boolean values that allow developers to use a new, experimental process on their application by gating it behind a virtual on/off switch, and have the option to disable it if a problem is found, all without another code deployment cycle. An example could be a “down for maintenance” notice, or something more advanced like a new product homepage that doesn’t require a code rollout to update the content.
Feature rollouts take this concept a step closer to real experimentation by placing users into two or more experimentation groups, allowing every new feature to be beta-tested before a full general release. Each new feature becomes an opportunity to run an experiment and collect immediate feedback on your development hypothesis.
Finally, there’s traffic splitting, a concept similar to feature rollouts that transparently routes a small number of users to a new experimental version to identify any bugs before they reach a broader audience. The primary difference between feature rollouts and traffic splitting is the “all-in” nature of traffic splitting an entire application vs. a single feature.
Worried that the automated and manual testing of your large product release didn’t catch everything? Send a small percentage of your traffic to the new application as a “canary build” and see what happens. Optimizely experiments with traffic splitting between different production environments (stable versus release candidate, for example) to a small number of users. Facebook, and others have utilized this process for years.
Shift Left, and Enter a High-Velocity Future
Thorough testing helps companies catch many issues before they reach customers, but it’s extraordinarily difficult to anticipate every possible usage or scenario for your product. Investing in these release strategies will help companies mitigate the risks of unintended bugs creeping into code or, worse, unwittingly turning their customers into the QA team.
Our advice? Don’t let feature development take priority over the necessary steps to keep your organization from working lean. Empower your developers to participate in the process early by shifting left, and watch developer productivity and velocity soar.
About the Author / Brian Lucas
Brian Lucas is Senior Software Engineer at Optimizely, responsible for the Build, Test and Release Engineering. Previously, he was CTO at Credible and Hoopla, and lead architect at Sling (Slingbox). He holds four patents. Connect with him on LinkedIn and Twitter.
