Question about Data Amount

[alvatore-code]

Hi Meridian team,

https://developers.google.com/meridian/docs/pre-modeling/amount-data-needed

I've read this document. However, i am still confused about how much data we need.

1. Is there a recommended ratio between the number of media channels and data volume for national models?
   While I understand that Bayesian inference technically doesn't have a strict minimum data requirement due to the regularization from priors, the documentation explicitly emphasizes having enough data for the number of parameters being estimated. From a practical standpoint—to ensure that our ROI estimates are actually data-driven rather than heavily dominated by priors—is there a general rule of thumb or recommended mathematical ratio (e.g., number of time periods / number of media channels) for national models?

2. How to quantify the benefit of information sharing in geo-level models before running the model?
   The documentation mentions that geo-level modeling allows for information sharing across different regions. Since we want to evaluate if our current geo-level data granularity is sufficient before investing time in running MCMC sampling, is there a heuristic, formula, or specific method to estimate how much this sharing actually helps effectively increase the data volume beforehand?

[cpulavarthi]

Hello @asong-code,

Thank you for contacting us!

To start, you might find this previous discussion on data requirements for Meridian models helpful: GitHub#1391.

To address your specific questions:

1. In Meridian, each media channel, non-media treatment, organic media, knot and control variable is considered an “effect”. Every effect has a parameter that needs to be estimated during model training. The ratio “Data points per effect” gives us an estimate of whether we have sufficient data. The rule of thumb estimates for data requirement aren’t fixed as they heavily depend on data quality and the signal to noise ratio in the dataset.

   • National Models: We usually suggest that ~15 data points per effect allows for directional information, whereas ~4 is considered a low sample-size scenario.
   • Geo-level Models: Due to partial pooling (information sharing across geos), the effective sample size is higher than the raw calculation. A lower raw ratio (e.g., ~8 data points per effect) can be viable for Geo models because the hierarchy allows the model to "borrow strength" across locations. (See Amount of data needed for more info).

   Data quality also impacts the amount of data needed per effect. How many data points are required per effect depends on your specific dataset and you can detect when the data you have is insufficient by performing post-modeling checks. Lack of prior-posterior shift is one such check. For this you can experiment with different sets of uninformative priors and see if the posteriors are converging consistently. If the posteriors obtained are dependent on priors, this suggests that the data is insufficient or doesn't have a strong enough signal in it.

   As a general rule of thumb, historical data should be a minimum of two years' worth of weekly data for geo-level models and three years' of data for national-level models. For geo-level models, a practical guideline is to use the top 50-100 geographical units (such as US DMAs) ranked by population size.

2. It is difficult to quantify the benefit of information sharing in a geo-level model before running the model. The benefit is dependent on the similarity of the media impact mechanisms across geos and the differences in media spend in each of these geos. The benefits are generally in terms of improvement in the confidence of model predicted estimates, so only the comparison of the post-modeling metrics between the corresponding geo and national-level models can quantify the benefit of running a geo-level model.

Feel free to reach out if you have any further questions or suggestions for us.

Google Meridian Support Team

[KatatoniaSh]

Hello
I have a question. For MMM, we usually have 3 years of weekly data (156 examples). To have ~15 points per effect, we need to have very few variables. For example, even if we consider the number of knots, in many cases, the number of knots determined by auto knot selection is much higher than that. For example, when I use auto knot selection, the number of knots is around 20 to 30 depending on the kpi. Based on my experiments, I guess we need to have 10-15 knots which results in small data points to effects ratio.
One more question, if the posterior depends on the priors, how can we determine the best priors given that we did not have any experiments before.

[cpulavarthi]

Hello @KatatoniaSh,

Thank you for your questions!

Knots in National Models

The default number of knots for a national-level model is 1. We recommend starting with 1 knot and increasing it only to capture major seasonal changes, rather than fitting all minor fluctuations. Each knot has its own regression coefficient that needs to be estimated. If Automatic Knot Selection (AKS) selects 20-30 knots, it is likely overfitting due to the limited data per time period in a national model. In this case, we would recommend manually choosing and setting the knots parameter rather than relying on the default AKS. You may reference our documentation on setting knots for further practical guidance in working with manual adjustment of knots and tuning the AKS parameter.

Determining Priors Without Experiments

If you do not have experiment data, you can inform your priors using industry benchmarks, results from previous MMMs, or your team's subject matter expertise (e.g., knowing that an ROI above a certain threshold is highly unlikely in your industry).

If you have no prior knowledge at all to guide the selection, we recommend using Meridian's default priors or other uninformative priors to check the consistency of model convergence. The default ROI priors are designed to provide a reasonable degree of regularization and to treat all media channels equitably. Ideally, the model should converge towards similar posterior distributions independent of the choice of priors as long as the priors chosen are fairly uninformative (wide priors with high standard deviation). If the posteriors are dependent on priors despite choosing uninformative priors, then it would suggest insufficient data or low signal to noise ratio in the data and you will have to perform exploratory analysis to identify and resolve any data quality issues you may be having in your MMM study.

Feel free to reach out if you have any further questions or suggestions for us!

Google Meridian Support Team