10 Red pill facts in Marketing Mix Modeling (MMM)
Facts learnt from building and commissioning 100s of MMM models
Hello Folks,
Both Ridhima and I have built 100s of MMM models and also commissioned them into practical use. As Aryma Labs, we have built over 300 MMM models in the past 5 years.
Here are some of the facts about MMM:
1) High spend on a marketing / media channel is a good feature selection criteria.
High spend on any marketing/media channel across many years is often the biggest signal that it has worked. This encodes institutional knowledge as well.
2) Though crude, the simplest validation of MMM is to see if the predicted KPI is matching the ground truth (once realized).
If your predictions match the unseen data or the future sales value, then you can be reasonably assured that the mix you got in your MMM model indeed is the one that lead to the sales you observed.
For further validation, techniques like DID is always available and recommended.
3) If a client is doing MMM for the first time, adopting Bayesian MMM is similar to blindfolding oneself and trying to take a swing at piñata.
You miss more than hit :)
4) Every Bayesian MMM vendor's north star is a secretly built Frequentist MMM or past Frequentist MMM model.
5) No matter what Bayesian MMM vendors proclaim, new information can't be encoded into MMM through priors.
Most Bayesian MMM vendors stick to using half normal as a prior because of the fear that the model could show negative signs for key variables.
This is no way reflects the underlying probability distribution the data or variables follow.
Further, most Bayesian MMM don't take into cognizance convolutions. Imagine concentric waves clashing into each other when you throw stones into a pond. Now just imagine multiple such concentric waves and you being tasked to disentangle the effect of each. That is what happens under the hood of a Bayesian MMM. All the 'we update our belief based on new evidence' is just marketing talk. Real Bayesian analysis is very tough to get right even for the most seasoned statisticians.
6) Most MMMs are built with an expectation to vindicate past or current marketing strategies.
7) The biggest incrementality test you can run is MMM. Yes, MMM itself is an incrementality test.
8) The more your KPI has human element (human error in data collection), the more wrong the MMM model will be.
For e.g. if your KPI is sales, the chances of model inaccuracy is low. If your KPI is mind measure metrics like TOMA - the chances of inaccuracy increases (see link in comments).
9) In terms of attribution accuracy and holistic representation of marketing reality:
MMM > Incrementality Tests/Experimentations > MTA (probabilistic) > Last touch First touch.
10) MMM's can never be fully automated aka Flintstones curse (See link in comments)
