Three Reasons Why Traditional A/B Testing is Dying

Everybody has seen the classic testing and optimization wheel on some smart art-heavy marketing PowerPoint slide. It seems so simple right? Start with a hypothesis, run the test, generate insights, and iterate. Wash, rinse, and repeat.

This model works great—especially if your customers only access your site or app via one device via one source on a linear transactional journey.

Oh wait, your business isn’t that simple?

From mobile applications and notifications to new social channels and IoT/wearables, consumers now have dozens of daily options for directly interacting with a brand. In the same vein, marketers now have thousands of highly contextual behavioral data points they might leverage to personalize experiences, along with an increase in offerings and channels with which to reach the consumer.
With these shifts, the number of potential tests has exploded, and a static A/B testing program no longer has enough velocity in a customer-centric paradigm.

If your A/B testing program is having problems keeping up with the speed of today’s consumer and your business goals, you’re not alone. Some of the world’s largest consumer brands still struggle to run more than a few meaningful A/B tests per month, and even when they do, they’re not nearly effective as they should be.

But why is that?

1. Manual setup and upstream segmentation creates guessing game for marketers
Traditional A/B testing and optimization programs require marketers to guess—err, I mean hypothesize—all the features of a test ahead of time. This means also choosing specific segments to target for the test. Should our re-engagement test of variations A/B/C go to people that haven’t engaged in 7 days? In 10 days? What about people who did recently engage, but that engagement was browsing my FAQs on how to cancel their subscription?

With this method, each hypothesis to be tested requires manual setup – or frequently an office spat over which tests/hypotheses to run given bottlenecks in execution.

2. Waiting for static winners wastes lift opportunities
Statistical significance is a beautiful thing. However, waiting for a winner can severely slow down your potential lift and hinder overall program performance. Machine-learning adaptive experimentation utilizing a multi-armed bandit approach is able to take earlier action to increase volume to the winners as soon as they begin to emerge—while minimizing opportunity cost of delivering lower performing experiences.

Given the technical implementation challenges plaguing most testing programs, it’s rare that you’ll find static experimentation moving at the speed of the modern consumer.

3. Over-generalizing results leaves gaps in context
With static experimentation, it’s normal to overgeneralize results based on one type of interaction at one given time (i.e. Hero A is the winner in one experiment, so everyone gets Hero A moving forward, regardless of a particular users current context or history of engagement with the brand). Today’s consumers and their hundreds of micro-interactions leave highly-contextual behaviors which traditional A/B testing can’t explore:

  • Is the winning offer the best for everyone? Would another message have worked better for customers who behave differently?
  • Is there another incentive that would have worked better in different contexts?
  • Would delivering the offer via a different channel or in a different sequence have been better for some customers?

With machine-learning enabled adaptive experimentation, marketers are able to run thousands of ongoing customer interaction experiments simultaneously to ensure the optimal message is delivered to the optimal user at the optimal time. This allows marketers to not only discover and exploit winning combinations of messaging and users, but it also allows opportunity to continue to explore for additional lift in the ongoing testing of new contexts.

As consumers continue to demand increasingly relevant personalized experiences, traditional static experimentation will continue to give way to powerful machine learning-powered platforms that leverages digital intelligence across nearly every customer interaction. You need a digital intelligence system that both tests and adapts.

Those that don’t will still be stuck in limbo waiting for statistical significance—one hand-crafted segment at a time.

About Garrett
As head of product for Amplero, Garrett dwells at the forefront of the machine learning-powered marketing revolution. Building the Amplero Digital Intelligence Platform since its inception, Garrett is a SaaS product leader focused on marketing transformation with deep experience across both MarTech and AdTech industries.

With more than a decade focused on product innovation, he’s built forward-focused technology and analytical solutions for brands such as XBox, Sprint, American Express, Citibank, and Ganett. Prior to Amplero, he held product leadership roles at Microsoft Advertising, AdReady, and aQuantive.

Contact him directly at

About Amplero
Amplero is a digital intelligence platform that enables marketers to achieve what's not humanly possible by leveraging machine learning and multi-armed bandit experimentation to automatically optimize every customer interaction to maximize customer lifetime value and loyalty.

Using Amplero, marketers in telecom, banking and finance, gaming, and software-as-a-service have seen more than three percent incremental growth in customer revenue and five times retention benefit, often touching their customers less frequently while delivering great omni-channel customer experiences.

To learn more or schedule a demo, contact us today.