The Right Way to Read MMM and Experiment Numbers Together

Spoilers: Calibrating MMM with Experiments is still a bad idea

After my post on “Why Feeding Experiment Results Back into MMM is the Biggest Scam in Marketing Measurement”, I got a lot of DMs.

One of them asked a very practical question:

“Ok, Venkat we can’t calibrate MMM with Experiments. But what decision would you take basis an MMM on an experimental result?”

My answer:
📉 Downsize the number from an Experiment in MOST CASES.
📈 Increase the number from an Experiment in SPECIAL CASES.

Here is my rationale:

When and why to Downsize the Experiment Number

In MMM, the overall effect on sales is captured by all the variables. All variables tussle with each other to claim their rightful ‘attribution’ for the sales.

But in an experiment, the media/marketing channel in consideration has no competition.
It is not tussling with any other fellow media channels. Hence the number (lift % or avg sales) in an experiment will always be inflated.

When and why to Increase the Experiment Number

Like with anything, there are exceptions.

Suppose your MMM includes interaction effects - say ASC and PMax campaigns working together.

Now imagine you run two separate experiments: one for Meta, one for Google.

Each experiment misses the synergy between them, so both effects will appear muted compared to MMM’s result of Meta contribution (Meta’s own contribution + the synergistic effect of ASC+Pmax)). Same applies to Google contribution too.

In such cases, it makes sense to slightly increase the experiment number.

Key Caveats

Remember that an experiment can never be the arbitrator of truth of Holistic Marketing Effectiveness. Because by design it doesn’t measure all variables effect at once.

Hence don’t use Experiments to calibrate MMM.

Use Experiments to directionally validate your MMM, not to overwrite it !!

And use MMM as a North Star to contextualize experiments within a short time window.

An MMM built from Jan 2024 – Oct 2025 can help you interpret experiments done in Oct or early Nov, but beyond that, it must be refreshed with the latest data to remain a reliable compass.

Because even MMM model drifts.

---

Thanks for reading.

For help with Causal Marketing Experiments, Experimentation and MMM, get in touch with us.