A/B Testing

A/B/N and Multi-variant testing is effective...but simply not scalable

Typical A/B Testing Interface

Leading companies have been leveraging A/B/N or Multi-variant testing to optimize dozens of variables since the late 1990's. However, in today's world, there are literally 100,000 or even 1,000,000+ permutations possible. While these testing efforts often lead to better results, most large enterprises can only run a few dozen tests per year.

One of the most common ways marketers try to determine the best marketing tactics is through A/B or multi-variant testing. With A/B testing marketers are focused on identifying the “winner” among a hypothesized set of potentially-good marketing interactions, targeted at certain customers and delivered in certain execution contexts.

The test identifies the offer that performs best in terms of take rate or other KPI, and therefore, should be launched broadly. But with each test there are questions:

  • Is the winning offer the best for everyone? Would another message have worked better for customers who behave differently?
  • Is there another incentive that would have worked better in different contexts?
  • Would delivering the offer via a different channel or in a different sequence have been better for some customers?

A/B testing can be better than guessing, but is limited to what can be hypothesized and constructed by the marketer. Because A/B tests are designed and executed manually and rely on the marketer to design the clean measurement of each test, identifying the winner for each customer in every context across a customer base of millions becomes impossible. Even when a marketer does find something that seems to work for the majority of customers, it may take weeks or even months to roll out that “winner” to all of their customer base. While the basic premise of A/B testing is a good one, this entire approach is trying to solve a problem with in a linear fashion (drive 2-5x scale vs. the current approach), but today’s connected customers require an exponential (1000x) approach. In short, the market is ripe for major disruptio​n.

Learn how Amplero can help your marketers leverage machine learning and adaptive experimentation to achieve what's not humanly possible. Request a live demo.