The Golden Period to do Experimentation
Hint: After completion of MMM Project
Is There a "Golden Period" for Experimentation?
Yes.
But when?
TL;DR : Right after completion of MMM.
But what is the rationale behind this?
Lets dive in.
Don’t Calibrate MMM models with Experiments, Use Experiments to Validate MMM
Many still believe that Marketing Mix Modeling (MMM) can be calibrated using experiments. However, this belief is not only incorrect but also akin to putting the cart before the horse.
MMM is a holistic approach that closely mirrors marketing reality, whereas experiments- such as lift studies, geo tests, or quasi-causal tests are more limited in scope.
MMM is a holistic approach that closely mirrors marketing reality, whereas experiments- such as lift studies, geo tests, or quasi-causal tests are more limited in scope.
That said, experimentation is incredibly useful in the following scenarios:
When you need a directional read on marketing attribution in the absence of MMM.
When you want to validate MMM findings.
Operating in the Shadows: The Best Time for Experimentation
The golden period to conduct experiments for proving marketing effectiveness is right after completing an MMM project.
Much like a shadow is cast during an eclipse, we believe MMM too casts a 'positive shadow'. MMM results tend to hold good for at least 3 months in the future if properly specified and the domain does not experience erratic changes.
Much like a shadow is cast during an eclipse, we believe MMM too casts a 'positive shadow'. MMM results tend to hold good for at least 3 months in the future if properly specified and the domain does not experience erratic changes.
Given that the model results hold good for 3 months into the future, here is why experimentations can / should be done in this period:
The ROI from experiments will never exactly match MMM ROIs due to time period mismatches, variable differences, and the absence of adstock effects in experiments. However, when operating within this 'Umbra period' (0-3 months post MMM), experiment ROIs are more likely to align with MMM results.
If there are discrepancies, they can be adjusted. For e.g. experiment ROIs can be scaled (e.g. by a factor of 1 or 2), based on MMM, since the holistic MMM model remains reliable during this time.
Coming to the 'Penumbra Period', MMM results start to degrade. From our experience, model inference and predictive quality deteriorate around 3-4 months post-build. This is why frequent MMM updates are recommended.
Experiments conducted in the penumbra period have a higher risk of divergence from MMM results. However, with proper checks and adjustments, they can still provide useful insights.
Some Caveats:
⚠️ Experiments and MMM should measure the same KPI.
⚠️ The variable tested in the experiment must be part of the MMM model.
⚠️ The same or similar quantum of spends seen during the MMM model period should hold during the experimentation period as well.
How you the marketer benefit from this?
Many marketers hesitate to refresh their MMM models, assuming it’s as costly as building a new one from scratch. At Aryma Labs, we take a different approach - our mid-tier pricing plan includes model refresh costs for the entire year.
Nevertheless, conducting experiments during the umbra period (0-3 months post-MMM) allows marketers to continue measuring marketing effectiveness accurately without immediately rebuilding their MMM.
In summary, MMM + Experimentation can be a powerful tool for marketers to have 'Always On Marketing Measurement System'.
Thanks for reading.
For consulting and help with MMM implementation, Click here