A sure sign of a company’s successful digital transformation is when it can leverage data to optimize every area within the enterprise. As Forrester puts it, for a company to realize the return on investment (ROI) of a digital investment, “benefits from digital investments need to be aligned with the business requirements for a specific value chain stage.” All along, data-driven insights need to be tied to measurable gains and real-world business key performance indicators (KPIs).
The reality, however, is that data-centric principles are easier to apply in some areas of the business than others. The ethos has certainly taken hold in the development world—with its whole culture around DevOps, continuous delivery (CD), continuous integration (CI) and related agile methodologies that emphasize constant, data-driven experimentation and frequent reassessment and adaptation of plans.
Unfortunately, too many companies drop the innovative mindset once things progress to production and customer rollout phases. In doing so, they fail to spread the ROI from digital investments more fully across the organization. In other words, they fail to mount a genuine digital transformation.
True digital transformation requires companies apply a stronger, more agile approach to product adjustments across the value chain. Nowhere is this more critical than the need to embrace “full-stack experimentation” to realize business value from the world of data-based product development and testing.
The Limits of A/B Testing
Especially in an era when the customer experience sits at the center of the value proposition for most industries, the ability to fine-tune everything about how we bring products and services to market—and how our customers use and feel about those products and services—is mission-critical, especially in the quest for true digital transformation.
Unfortunately, no matter how agile and data-driven the development phase may initially be, we lose visibility, insight and, most importantly, revenue whenever we fail to infuse those same values into the later stages of production, QA and analysis around how customers are actually using what they buy.
Perhaps the biggest sign of this is the over-reliance of marketing teams on signals from basic A/B testing.
Industrywide, A/B testing is the most common approach to experimentation, but it’s far from ideal. A/B testing is typically used by marketing teams to test front-end changes at the user interface-level. It’s limited in its efficacy, as it does not take into consideration the deeper statistical nuances and resultant effects across the application stack—for instance, swapping out code to roll out an entirely new feature to a small set of users and measuring the results of that single deployment across both business and system metrics.
Worse yet is the false sense of security that can come from the narrow scope of A/B testing. Any outcome can be easily misunderstood as the only improvement possible. That leaves countless other variables and opportunities for ROI unexamined. If Plato remarked, “The unexamined life is not worth living,” what does that say about the unexamined product development process? How do we know if it’s worth building?
A Better Way
While A/B testing is still appropriate for some limited uses, leading-edge companies are learning to expand inquiry across the “full stack” of their applications. Full-stack experimentation differs from traditional A/B testing in scope, purpose and the potential impact on a business. It examines not just user interfaces but also execution logic, technical services, APIs and the data infrastructure underlying it all. This fosters a market-driven approach that brings more insight and less guesswork to decision-making.
To be clear, full-stack experimentation does not optimize part of a product or feature; it is about optimizing the process around experimentation and allowing development to more directly support and inform the evolution of the business. In this way, full-stack experimentation helps gain deeper value from the organization’s use of agile, DevOps and CI/CD so we can garner critical insights into how a product or feature is actually used by customers. More importantly, it measures the impact of every release on important key business metrics—which may be radically different from the developer’s intent.
Full-stack experimentation unlocks value across the entire data ecosystem. It involves collaboration among the key stakeholders involved in the development of a product or application, from engineers to product managers, as opposed to primarily marketers. You can use full-stack experimentation to answer deeper questions and test deeper assumptions. For instance, does adding the ability to tag friends in a photo increase their likelihood of posting photos in the future? Does this result in more sessions per user and does that ultimately impact the ad revenue generated per user?
Delivering Digital Transformation to More Companies
The benefits of full-stack experimentation are well-known among organizations with the resources to build and manage this in-house. Microsoft, Etsy and Netflix, for instance, test everything from multiple angles, sometimes running 50 tests a day or more on a single feature. These companies have powerful instrumentation to capture and analyze the multitude of variables and metrics—including those that may have been unanticipated. This surfaces valuable insights to be quickly spotted and harnessed for value.
Fortunately, modern software-as-a-service (SaaS) solutions are increasingly saving companies from having to create their own in-house, custom-built systems to get the benefits of full-stack experimentation. Unified platforms now exist that help engineering and product teams accelerate the pace of product delivery and make data-driven decisions without investing valuable development resources to build the tools themselves. Critically, these platforms have robust feature flagging and extensive experimentation capabilities that allow testing to happen well into production and rollout to customers—with strategic and incremental tests to examine and adjust logic, APIs, UX or other elements across the entire application ecosystem.
On a tactical level, these new platforms make experimentation processes simpler and more accessible to more companies. But the bigger picture is that these new options for full-stack experimentation effectively remove the friction that exists in achieving true digital transformation—allowing more companies to completely revolutionize how they do business.