Why Feeding Experiment Results Back into MMM is the Biggest Scam in Marketing Measurement
Use Experiments to validate MMM and not to calibrate it.
MMM is great and so are Experiments.
One is designed to answer the holistic marketing effectiveness question and the other is designed to answer the impact of individual channels and
marketing campaigns.
But mixing both is like mixing oil and water.
Recently, a brand head evaluating our solutions asked: “Do you feed experiment results back into your MMM like some vendors do?”
Our answer - No, we don’t believe in calibrating MMM models with experiments.
It is for both ethical and statistical reasons.
The Statistical Reasons
The following is taken directly from our research paper “Why you should not calibrate MMM models through experiments” (which conclusively proved that calibrating MMM with experiments worsens the model). You can read the summary in our MMMGPT -> Summarize research paper. (Link in comments.)
▪️ Temporal Mismatch
Experiments, such as Geo hold out tests, typically span short durations (e.g weeks or months) and focus on immediate outcomes.
MMMs, on the other hand, are designed to analyze long-term, cumulative effects of marketing activities over extended periods (e.g. years).
This temporal mismatch makes experimental data unsuitable for accurately calibrating MMMs.
▪️Univariate Focus:
Marketing Experiments often evaluate the impact of a single marketing channel in isolation.
MMMs are multi-dimensional and account for interactions between multiple channels (e.g., TV, digital, and out-of-home advertising), as well as external factors like seasonality and economic trends.
Ignoring these interactions leads to an incomplete understanding of marketing effectiveness.
▪️Measurement Disparities:
Experimental data outputs (e.g., lift percentages or Average Treatment Effects) often differ in units and scale from MMM (e.g., revenue or sales).
Combining these data types directly can distort model parameters and result in biased predictions.
Why some vendors calibrate MMM with Experiments (Hint: to achieve flywheel effect).
Honestly speaking, feeding experiment results back into MMM serves well the vendors and not you the customers. Vendors would stand to gain because now there is flywheel effect created.
Run MMM → Claim “it’s inaccurate” → Run experiments (as a fix for MMM) → Feed Results Back → MMM becomes worse → Redo MMM → Claim MMM is inaccurate again, run experiments to fix it.
The cycle continues and you the client keep paying without getting any accurate insights.
At the end of the day, you still don’t have to take my word for it.
Hire any top statistician. Ask them if a multi linear regression coefficient should be changed based on an experiment results like Geo hold out.
Use Experiments to validate MMM and not to calibrate it.
Related articles:
Why you shouldn’t use Geo tests to fix priors in Bayesian MMM - https://open.substack.com/pub/arymalabs/p/why-you-shouldnt-use-geo-tests-to?r=2p7455&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false
The Golden Period to do Experimentation - https://open.substack.com/pub/arymalabs/p/the-golden-period-to-do-experimentation?r=2p7455&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false
Thanks for reading.
For help with Causal Marketing Experiments, Experimentation and MMM, get in touch with us.
 | A guest post by
|