Applying Media Quality Inside A Bespoke MMM: Six Tasks

At Adelaide we have found that our AU media quality data improves MMM output granularity and actionability but the uniqueness of each MMM means each brandamp8217s ability to apply AU within its particular MMM context can vary There are industry efforts underway to bring some standardization to MMM creation from the IAB and CIMMThis article describes six AUbased tasks a brand can execute to answer business questions in the context of its bespoke MMM Each task addresses a different strategic question The tasks can inform and are informed by the brandamp8217s MMMSix TasksAll the tasks listed below make bespoke MMM outputs more actionable and therefore more valuable for media teams at brands and agenciesTask A Validate the MMM against AU an external data sourceTask B Monitor AU media quality levels between MMM refreshesTask C Constrain the MMMamp8217s scenarioplanner with AU media quality thresholdsTask D Use AU to reallocate inside a channel before shifting budget outTask E Assess MMM outputs based on AUweighted impression inputsTask F Discover planning thresholds by splitting aggregated channel inputs into AUscored groupsWhySource Adelaide Metrics illustrativeThe chart above is the premise for all six tasks An MMM usually reports media channel effects at an aggregate level MMMs donamp8217t typically report media channel effect ranges inside channel aggregates but those ranges are criticalamp160 Analyzing them with AU data to understand an MMMamp8217s ingoing media quality levels and the ensuing impact of those levels is the first step for all the tasks described belowThe tasks are ordered by the effort required to execute them from the brandamp8217s analytics and media teams and its MMM provider Most brands end up carrying out several of the tasks on the listTwo CaveatsCoverage AU scoring depends on measurement access which varies by channel and by publisher Adelaide can typically score 90 of media on a plan but coverage should be doublechecked ahead of each MMM refresh cycleUnmeasured impressions Document how unmeasured impressions will be treated Options are via an Adelaide channel norm via the brandamp8217s own measured average for that channel and partner or via labeling unmeasured inventory as its own variableTask A Validate The MMM Against AU An External Data SourceMMMs are largely validated against themselves Holdout error coefficient stability and posterior checks are all useful but each is downstream of the same data and the same assumptions used to produce the MMM in the first place Independent corroboration of MMM findings is harder to come by AU is built from completely different data inputs and trained on independentlyproduced outcomes data MMM and AU inputs have no overlapsWhat the task answers Do the MMMamp8217s channel results agree with an external gauge of media qualityamp8226 Rank media channels and partners in an MMM against their effectiveness measureamp8226 Rank the same list of channels and partners by average AUamp8226 Compare the two lists and assess any differences in rank positionWhere the rank positions agree the brand has outside confirmation of both sourcesamp8217 validity Where they disagree the brand can treat that as a finding and look for a cause which could be a channel the MMM cannot isolate a gap in AU coverage or a lowerlevel difference in media quality the MMM does not detectWhatamp8217s needed the MMMamp8217s published channel or partner estimates and AU scores for the same periodSource Adelaide Metrics illustrativeTask B Monitor AU Media Quality Between MMM RefreshesMedia quality levels can go up or down between MMM refreshes supply paths change partners substitute placements seasonal inventory changes A channel can deliver the same impressions at the same cost and score lower on AU than it did when the MMM was built So if a channel misses its expected effect check media quality level changes on specific inventory first amp8211 it is the easiest data point to test and usually the easiest to correct requiring just a partner conversation rather than a budget or plan changeWhat the task answers The MMMamp8217s recommendations are a year old Is the media being bought today still the media the MMM was fitted onamp8226 Track the average AU of each channel on an ongoing basis against that of the time period the MMM was fitted onamp8226 Set an AU threshold a few points below the fitted averageamp8226 Treat a threshold breach as an alert to examine supply paths and partner mixThis task can be executed as an ongoing monitor or as a spot testWhatamp8217s needed ongoing AU scoring and the MMMamp8217s channel estimatesTask C Constrain The MMMamp8217s ScenarioPlanner With AU Media Quality ThresholdsEvery MMMprovided scenarioplanner tool assumes that the next impression in a channel will look like the average impression the MMM was fitted on but that assumption is probably untrue Since a brand probably already buys its most desired inventory adding more inventory in a specific channel requires buying further down its own supply curve amp8211 so the scenarioplanneramp8217s projected effectiveness measure when adding budget to a channel is probably optimisticWhat the task answers The MMM says to put another 2 million into CTV Can the buying team execute thatamp8226Derive an AU threshold from the brandamp8217s own outcome data join AUscored exposures to media the brand already tracks exclude media below a threshold step by step and remeasure the outcome variable at each step Where the outcome performance measure inflects is the media quality thresholdamp8226 Run the same analysis separately per outcome variable per media channel amp8211 a display threshold and a CTV threshold are not comparableamp8226 Then add the media quality threshold values into the scenarioplanneramp8217s constraint settingsThus constrained the scenarioplanner output can be amp8220another 2 million into CTV at or above 44 AUamp8221 an instruction which can go on an insertion order or into a PMP deal term or programmatic bidding ruleWhatamp8217s needed AUscored exposures joined to outcome data a rollingthreshold analysis and the MMM scenarioplanner outputTask D Use AU To Reallocate Inside A Channel Before Shifting Budget OutBefore acting on channel budgetreduction recommendations dissect that channelamp8217s delivery by AU levels and assess the brandamp8217s outcome variable performance data within the channel rather than against other channels If the high point of the channel range performs well on outcome effects but the bottom of the range does not the budget recommendation can become amp8220reallocate placements in this channelamp8221 instead of amp8220spend less in this channelamp8221What the task answers The MMM says to cut display Does it mean cut all display or just cut the lowestquality displayamp9675 Identify highquality placements with AU qCPM amp8211 the cost divided by impressions at or above the AU threshold per thousandamp9675 A publisher with a 6 CPM delivering 22 of its impressions above the AU threshold produces an AU qCPM of 2727 a publisher with an 11 CPM delivering 66 above the AU threshold produces an AU qCPM of 1667amp9675 The cheaper CPM media is the more costly source of highquality impressionsThis task needs the AU threshold definition from Task C or Task F Itamp8217s the quickest of the six tasks to execute because the analysis can be done for a single channel by a single teamWhatamp8217s needed AU scores media cost data the MMMamp8217s channel verdict and a threshold from Task C or Task FSource Adelaide Metrics illustrativeTask E Assess MMM Outputs Based On AUWeighted Impression InputsAn impressionsbased MMM reports the impact of every impression as the same so channels that produce cheap reach look productive in proportion to volume and total media contribution to sales is reported well above a spendbased model output Qualityweighting media impression inputs with AU scores can help Weighted inputs are already compatible with most major opensource MMM platformsWhat the task answers Why does the MMM attribute so much of the outcome to the least expensive channelsamp8226 Weight each impression by its AU score before loading it into the MMM then rerun the analysisamp8226 Compare the AUweighted analysis findings against the unweighted analysis findings on three dimensions holdout fit coefficient stability across refreshes and whether total media contribution to sales is more plausible when considered against the brandamp8217s spendbased benchmarkWhatamp8217s needed an MMM provider one revised input file one additional MMM runTask F Discover Planning Thresholds By Splitting Aggregated Channel Inputs Into AUScored GroupsA media quality threshold established from outcome data is the strongest input available Brand lift studies conversion data joined to AUscored exposures and pooled results across campaigns all produce a defensible AU minimum tied to a named KPIWhen data is missing because studies were not run outcome data does not join cleanly or the brand is new to media quality measurement the MMM itself can produce the media quality thresholdWhat the task answers With no outcome study to draw on can the MMM itself produce a planning thresholdamp8226 Bin media impression volume by qualityamp9675 Divide each channelamp8217s impressions into three or four nonoverlapping AU bands for example 0 to 30 30 to 45 45 to 60 and 60 and aboveamp9675 Every impression falls into exactly one band The bands still sum to the channelamp8217s original volume so nothing is added to or removed from the MMM media data inputsamp8226 Load the bin impression volumes as separate model inputsamp9675 Each band becomes its own media variable carrying its own coefficient carryover term and response curve The MMMamp8217s structure is otherwise unchangedamp8226 Measure the effect each bin contributesamp9675 Observe effectiveness changes as media quality rises The point at which the coefficients step upward is the brandamp8217s threshold derived from its own business outcomeamp9675 The check that the exercise worked is that the bands rankorder with AUamp8226 Weight and consolidate the bins according to what the MMM reportedamp9675 Bands that perform alike should be combined Bands that perform differently should carry the weights the MMM assigned rather than an assumed scaleamp9675 This step converts a diagnostic into new tools for a planner a threshold a qualityadjusted CPMWhatamp8217s needed AU scores on the channelamp8217s impressions an MMM provider a change to the MMM specification and a refresh window The most involved task on the listAll six tasks listed above refer to the same premise stated in the beginning brands can obtain more output granularity and critically more business applicability and actionability by using AU to disaggregate the media averages their MMMs are likely reporting Adding AU to a bespoke MMM is a great step for a brand seeking new ways to get value and business intelligence out of its MMM investmentCoauthored with Ben Lowe Adelaide Senior Director Client Strategy ampamp InsightsPosted at MediaVillage through the Thought Leadership selfpublishing platformClick the social buttons to share this story with colleagues and friendsThe opinions expressed here are the author39s views and do not necessarily represent the views of MediaVillageorgMyersBizNetamp160

Become a Thought Leader. Share your thought leadership article with industry professionals & have them archived in Thought Leadership

Applying Media Quality Inside A Bespoke MMM: Six Tasks

Author

Publish date

Channel

Claire Browne

Tim Spengler Read Your Third Brain. Here’s What He Wanted to Ask Jack Myers. - By Jack Myers

More Videos:

Become a Thought Leader. Share your thought leadership article with industry professionals & have them archived in Thought Leadership

Applying Media Quality Inside A Bespoke MMM: Six Tasks

Enjoying This Commentary? There's More to Love

Claire Browne

TAGS:

Tim Spengler Read Your Third Brain. Here’s What He Wanted to Ask Jack Myers. - By Jack Myers

The payment was successful

Cookies