Part 3: Why Legacy Analytics Tools Are Failing Marketers

The pain of running testing programs at scale

(Ed note: This is the third installment of a four-part series discussing how AI marketing technology disrupts the traditional analytics continuum with Amplero’s head of product Garrett Tenold. In “Part 1: Moving marketers out of neutral,” we covered the strengths and limitations of legacy analytics and business intelligence tools, as well as the current interplay between analytics and human input.

Part 2: Exploring the relationship between analytics and action” discussed how marketers can influence preferred action and understanding the difference between correlation and causation.)

If you’ve stuck with us this far, you know that behavioral optics, or even high-value predictive modeling, is not enough to deliver on the promise of prescriptive analytics.

Over the past decade, marketers have made considerable talent and infrastructure investments to support testing and personalization tools only to struggle in keeping pace with evolving nature of consumer behavior and the performance demands of the business.

In fact, some of the world’s largest consumer brands still struggle to run more than a few meaningful experiments per month, and even when they do, they’re not nearly effective as they should be.

But why is that?

Here's a common scenario:

The CRM team at a major B2C brand wants to migrate legacy users to a modern subscription plan, measuring success as the number of users who migrate to the new plan and stay for at least 30 days.

Imagine a world where the CRM team knew almost nothing about their customers and only wanted to run a simple experiment:


<Which Users>

  • Plan = Legacy 1 or Legacy

<What Marketing Intervention>

  • Discount = 10% or 20%

This results in 2^2 = 4 total tests with the following configuration.

What is impact (compared to controls) when:

  • Plan = Legacy 1 and Discount = 10%
  • Plan = Legacy 1 and Discount = 20%
  • Plan = Legacy 2 and Discount = 10%
  • Plan = Legacy 2 and Discount = 20%

But then they add one more user characteristic that they might know:

<Which Users>

  • Plan = Legacy 1 or Legacy 2

  • Current Usage = High or Low

<What Marketing Intervention>

  • Discount = 10% or 20%

The team is now up to 2^3 equaling 8 total tests. This is a lot more work to set up, but still completely manageable for a rockstar team.

What is impact (compared to controls) when:

  • Plan = Legacy 1 and Current Usage = Low and Discount = 10%
    • Plan = Legacy 1 and Current Usage = Low and Discount = 20%
    • Plan = Legacy 2 and Current Usage = Low and Discount = 10%
    • Plan = Legacy 2 and Current Usage = Low and Discount = 20%
    • Plan = Legacy 1 and Current Usage = High and Discount = 10%
    • Plan = Legacy 1 and Current Usage = High and Discount = 20%
    • Plan = Legacy 2 and Current Usage = High and Discount = 10%
    • Plan = Legacy 2 and Current Usage = High and Discount = 20%

However, as marketing teams know, the real world involves considerably more variables.

<Which Users>

  • Plan = Legacy 1 or Legacy 2
  • Current Usage = High or Low

<What Marketing Intervention>

  • Discount = 10% or 20%
  • Channel = Email or Push notification

Now, we’re at 16 tests to choose from for manual configuration.

This still is at a toy problem scale.


Any marketer will know recognize the reality looks far more like below:

<Which Users>

  • Plan = Legacy 1 or Legacy 2
  • Current Usage = High or Low
  • Hundreds of other behavioral dimensions available in your customer data ecosystem

<What Marketing Intervention>

  • Discount = 10% or 20%
  • Channel = Email or Push notification
  • Dozens of other characteristics of content, creative, tone, incentive types, incentive amounts, subject lines, time of day, and day of week

A typical B2C enterprise can have easily 2^100 to 2^1000 possible tests they might run. This may sound small, but the length of the entire universe in seconds is only 2^44. There’s simply no way to know which of these might bear fruit, and manually picking a few to test/automate will barely scratch the surface of ensuring that you’re optimally driving outcomes by matching the right marketing interventions to the right users.

Not only is the experimentation process slow and manual, but every new insight generates new work. While a full optimization team might run five tests in a month and generate statistically significant insights, taking actions on those insights involve rushing away to rewrite their targeting rules and update segment definitions. To achieve true customer intimacy, it requires holistic context of every relevant customer attribute and behavioral event, which requires the testing and insight rate to increase a thousand-fold.

Marketers simply can’t manually apply every micro-insight needed to drive true channel-agnostic personalization at scale.

Even if you successfully execute against your testing program, it’s nearly impossible to execute this approach across the entire cross-channel customer journey with traditional testing and rules-based journey builder tools.

In our next and final post, we’ll focus on building systems of action through artificial intelligence marketing (AIM) technologies to finally deliver on the promise of prescriptive analytics.

To learn more about building systems of action through AIM technologies, download the case study on how Amplero helped a leading mobile provider gain 650% lift in ROI, or schedule an AIM Platform demo for your team today.

About Garrett

As head of product for Amplero, Garrett dwells at the forefront of the machine learning-powered marketing revolution. Building the Amplero Digital Intelligence Platform since its inception, Garrett is a SaaS product leader focused on marketing transformation with deep experience across both MarTech and AdTech industries.

With more than a decade focused on product innovation, he’s built forward-focused technology and analytical solutions for brands such as XBox, Sprint, American Express, Citibank, and Ganett. Prior to Amplero, he held product leadership roles at Microsoft Advertising, AdReady, and aQuantive.

Contact him directly at gtenold@amplero.com.

About Amplero

Headquartered in Seattle, Amplero is an Artificial Intelligence Marketing (AIM) company that enables business-to-consumer (B2C) marketers at global brands to optimize customer lifetime value at a scale that is not humanly possible.

Unlike traditional rules-based marketing automation systems, Amplero’s Artificial Intelligence Marketing Platform leverages machine learning and multi-armed bandit experimentation to dynamically test thousands of permutations to adaptively optimize every customer interaction and maximize customer lifetime value and loyalty.

With Amplero, marketers in competitive, customer-obsessed industries like telecom, banking, gaming and consumer tech are currently seeing measurable lift across key performance indicators—including 1-3% incremental growth in customer topline revenue and 3-5x lift in retention rates.

For more information, contact us today or follow us on Twitter.