Which technique provides for greater manipulation in MMM - Bayesian or Frequentist?
How Bayesian MMM allows room for manipulation.
Hello Folks,
In our last few posts, we saw why Frequentist MMM is better. ICYMI, the following links should help you catch up.
In today’s post, we are going to discuss about manipulation in MMM. We know we might be doing an equivalent of ‘Breaking a Magician’s code’ and our fellow industry practitioners might hate us for this. But all we care about is providing absolute truth to you the readers, so that you can make informed decisions about MMM.
The monkey balancing
Some vendors will have you believe that both frequentist and bayesian MMM are wonderful and that one should not choose one over the other. Sorry to say, most people saying that don’t have in-depth grasp of statistics nor great knowledge of MMM. Conviction is normally borne out of deep knowledge and experience.
We at Aryma Labs have done in depth research on Bayesian MMM, have also applied it ourselves and also seen other vendors’ result of applying Bayesian MMM to clients.
So over the course of years, we have formed some deep convictions about Bayesian MMM backed by solid statistical proofs and empirical results.
Which technique allows for greater manipulation?
Ans: <drum roll……> Bayesian MMM.
How does Bayesian MMM allows for greater manipulation?
Short answer - Through priors and Posterior Distribution
It is well known fact that one of the Achilles heel of Bayesian technique is the subjective priors.
In a small data (relatively speaking) problem like MMM, one never has enough data where the evidence in data could overwhelm the priors.
This is hence a serious flaw in the technique. But this flaw could also be used in a nefarious way.
We have seen Bayesian MMM analysts confidently specify the location and scale parameters 'for example - the contribution from YouTube should be 7% or that the ROI of a Instagram ad spends should 3.4'.
Remember above that we mentioned there is not enough data to overwhelm the priors. This fact allows MMM analysts to 'fix' the contribution % or ROI numbers to their desire. (wink wink ;))
Why do they do this?
They do this because they think these numbers would make the clients happy.
The repercussion
The repercussion is that, you as a client won't have the true picture of your marketing effectiveness.
The next culprit: Posterior Distribution
In Bayesian methods, Posterior distribution is the final frontier. All the answer one seeks is in there.
The actions that bayesian MMM analysts take with respect to posterior distribution is quite befuddling, contradictory and paradoxical.
Bayesians deride frequentists that they rely on point estimates and can't quantify uncertainty (not true - will explain how in another post).
But when it comes to MMM, Bayesian analysts end up choosing a 'sort of' point estimate itself. They choose the expected value (mean) of the posterior distribution as the most likely contribution number of the particular marketing variable.
All the preaching of 'you don't quantify uncertainty' towards Frequentist takes a back seat. The contribution charts of both Frequentist MMM and Bayesian MMM now looks same.
If the mean of posterior distribution is not 'making sense' they can choose median or any percentile under the distribution.
You see a Bayesian MMM analyst hence has a lot of 'room' (technically the whole probability distribution itself) to select an 'ideal number'.
What about Frequentist MMM ? Does it not have levers for manipulation?
Frequentist MMM is not impervious to manipulation but it is several magnitude better than Bayesian MMM.
How?
It relies more on Maximum likelihood Estimation (MLE).
MLE basically tells you, given the data, what value of parameter is more likely to have generated the data.
Basically the data talks, not the analysts :) We call MLE the truth whisperer.
Because of the above reasons and more (stay tuned for more posts on this topic), we prefer to use Frequentist methods to fit our MMM models.
Thanks for reading.
For consulting and help with MMM implementation, Click here
Stay tuned for more articles on MMM.