Bayesian MMM inflates ROIs !!
You might be overspending on Performance Marketing and underspending on Brand Marketing, courtesy - Bayesian MMM
There is a popular saying "The cure must not be worse than the disease". In Bayesian MMM, this often becomes a reality.
We are going to purposely keep this post very statistics and math light so that every marketer gets the gravity of the problem.
The problem - Negative Signs
In MMM, negative ROIs or contributions are usually undesirable (except for competitor channels, macro factors, and seasonality)
To prevent this, Bayesian MMM analysts use probability distributions that never allow negatives - typically Half-Normal or Log-Normal distributions.
The half normal is your typical bell shaped curve but the left portion is cutoff which means that you only have positive values.
Next comes the Log normal. If you remember your high school math, Log is not defined for 0 or negative values.
Many Bayesian MMM vendors and opens source Bayesian MMM libraries like PYMC and Meridian use the above two methods.
The Posterior Distribution
Once a prior is defined in Bayesian models, the output is got from something known as posterior distribution. We will be skipping the likelihood part for the sake of brevity.
The posterior distribution is from where Bayesian MMM get all their answers.
For all the 'we are different from Frequentists and don't like point estimates' chest thumping, Bayesian MMM analysts become frequentists here.
You see quantifying uncertainty ≠ removing uncertainty. In reality, marketers want a clear go/no go answer. Not a ambiguous "your ROI could be between 4.5 to 6.4 and there is a 85% probability of it."
For all the 'we are different from Frequentists and don't like point estimates' chest thumping, Bayesian MMM analysts become frequentists here.
You see quantifying uncertainty ≠ removing uncertainty. In reality, marketers want a clear go/no go answer. Not a ambiguous "your ROI could be between 4.5 to 6.4 and there is a 85% probability of it."
Because we want clear go/no decisions, Bayesians have no other way but to provide point estimate from the posterior distribution !!
The Mean of Posterior vs the HDI of Posterior
Here is where things go wrong. In Log normal distribution, Bayesian MMM vendors give you the mean of the posterior as opposed to the HDI.
But you see from the image that log normal has this weird tilt which we statisticians call 'skewness'. Further a unique property of lognormal distribution is that the Mean is highest in comparison to mode, median and most of the times the HDI even !!
The Marketing Repercussions
okay enough of the statistics talk. What does all of the above mean to you the marketers?
From above image, we can see mean of log normal as ROI is > providing HDI or median or mode as ROI. This means you would be having a highly inflated ROIs.
In MMM, there is always a Yin and Yang force acting. If the contribution from marketing channel is more, the base (brand equity) will get smaller and vice versa.
When you take in inflated ROIs of media channels, you could severely also be underestimating Brand Marketing efforts !!
In turn this could mean that you might over spend on performance marketing channels as opposed to brand building channels.
The Key Takeaways:
As a marketer if you have roped in a Bayesian MMM vendor, ask these two important questions to them.
▪️Are you using log normal distribution?
▪️Do you take mean of lognormal instead of HDI to represent ROI and Contribution numbers?
Given all these complexities and problems with Bayesian MMM, we would highly recommend going with frequentist MMM. Less hassle, Less Complexity and most importantly you get realistic and accurate ROI numbers.
Thanks for reading.
For accurate MMM implementation, reach out to us