Debunking Misconceptions: The One True MMM Model Debate

Marketers need to understand Frequentist Statistics better

Earlier this week, MMM Labs tagged us in an attempt to refute our stance: There is only one true MMM model.

I engaged in a healthy debate in their LinkedIn post . I quickly understood the reason for their dissonance - Misconceptions about Frequentist Statistics.

1) Misinterpreting Confidence Interval

"Frequentist analysis will say, for example, 'the 70% confidence interval is [3%, 8%]. and we will also say 'too bad' if the future number falls outside of this interval. And the 95% interval may well include 0% contribution. So what is different here?"

Unfortunately, the above gives a Bayesian twist to a Frequentist concept.

- In Frequentist methods, the true parameter value exists and is fixed.
- The confidence interval (e.g. 95% CI) reflects coverage:

If you repeated the experiment 100 times taking different samples and constructed a confidence interval each time, 95% of the constructed intervals would capture the true parameter.

In frequentist methods we don't represent Confidence interval like '70% confidence interval is [3%, 8%]' .

Rather it is like TikTok contribution is 4.2 with CI [3.2,8.5] where CI could be 95%.

You see that in Frequentism we believe there is a true value of 4.2 (a point estimate) and this value either exists in the constructed CI or not.

There is no probability statement associated with it.

2) "Frequentist is a Special Case of Bayesian"

This claim made me smile. Bernstein von theorem states that under certain conditions, the Bayesian posterior distribution converges to a normal distribution centered at the maximum likelihood estimator (a frequentist approach).
You will find many literature talking about Bayesian convergence to frequentist results and not the other way round.

3) Parameters Are Fixed, Not Distributions

"In the frequentist approach, you assume that the estimated parameter is normally distributed." is incorrect.

In statistics, parameters are not normally distributed. The probability distribution could be. Parameters are fixed and have a value. Parameters characterize the distribution.

With these misunderstandings clarified, it is easier to see why some marketers struggle with our assertion. A correct understanding of Frequentist statistics illuminates why there is, in fact, one true MMM model.

Our endeavor is to foster informed debates and to push the field forward towards statistical accuracy.

---

Thanks for reading.

For consulting and help with MMM implementation, Click here

---