Most Experts Agree That Audience Duplication Needs Regular Evidence of Empirical Grounding

In the June 28 article we offered evidence that random probability methods do not accurately reproduce audience duplication patterns as measured empirically by best practices Nielsen ONE Ads represented the empirical approach and what used to be called Sainsbury the assumption of random probability duplication represented that one general modeling approachMany people became involved in discussing the article with me after publication and it became a single email thread involving over twenty leading technical minds in the media research industry One of the participants Advertising Research Foundation ARF Chief Research officer Paul Donato found the ad hoc collaboration exhilarating and decided to publish it Here is the link ARF discussionThe discussion exemplified what I had described as quotinclusive industry collaborationquot in my Hall Of Fame acceptance speech at Cynopsis last monthThe thread also shows that most media researchers would still prefer measurement to any form of modeling with an openness to modeling where measurement is not an option due to any specific walled gardenamp8217s insistenceThe discussion also revealed to many who did not already know it that walled gardens can be measured but only if they want to be What they then do is integrate with a chosen measurer either by direct match or through a clean room that preserves all the empirical relationships without risk to privacy Nielsen has been the typical integration choice of walled gardens so farAlthough the Virtual IDs approach embedded in the World Federation of Advertisers WFAAssociation of National Advertisers ANA CrossPlatform Audience Measurement Initiative is not identical with Sainsbury I opined in the article that both methods would tend to lack fidelity with the empirical duplication patterns But why did I say that a number of my colleagues wanted to knowHere was my underlying reasoning which did not appear in the article good thing too because if I had included my thinking to that degree the thread discussion might never have happenedOne Virtual ID is assigned to replace a given ID in one platform and to replace another ID in another platform The likelihood of them both actually happening to represent the same human being is low One of them may have been exposed to a brandamp8217s TV campaign and the other may not have been When the reachfrequency is put together across all media measured TV and digital are the starting point the arbitrariness of the VID assignments will have a randomizing effect with some VIDs found to have been exposed to TV and digital and other VIDs found to have been either unexposed or only exposed to TV or to digital but not both However the assignments have a good chance of not being accurateIn the first quotsketchquot many of these VIDs are likely to have been misclassified Some VIDs said to have been unreached by a platform might actually have been exposed to that platform and many said to have been reached might actually have not been reached by that platform The WFAANA model originally developed by Google and Facebook expects those errors and the idea is to then make iterative comparisons with a probability sample panel that is taken as the truth standard and the model machinelearns how to maximally force the model to agree with the truthsetThat might work if data were rigidly stable from week to week which was shown to be anything but the case Otherwise the trailing correction factors would be outdated by the time they are appliedMore importantly there are just so many things that can be controlled by weighting or changing simulated exposure statuses TRA found that an iterative algorithm could control for about 100 vectors and if we went much higher than that the model would not converge and we could wind up with even greater differences than we started with TRA was not trying to simulate audience duplication patterns among thousands of media vehicles across hundreds of popular target audiences we were merely trying to weight and project settopbox data to Universe Estimates UEs by demographics within 210 unduplicated markets in the US We found a geodemo set of 110 vectors that optimized the results for that situation and were never more than two percentage points off any UEsThis foregoing explanation of how I formed my opinion of the probable inaccuracy of the VID approach was the reasoning behind my statement in the June 28 articlequotWhereas Project Blueprint found that duplication among media are definitely not random the WFAANA blueprint allows for a concept of Virtual IDs which will tend to produce results that are very similar to random probabilityquotFormer Nielsen Chief Research Officer Mainak Mazumdar now Chief Research and Analytics Officer at Fox provided what was to me the most solid answer in the discussion when he reported that Nielsen in testing VIDs found them to be 90 accurate for duplication between two platforms but the more platforms the less accurate it became dropping to 75 when a third platform was added and down to 50 after thatThis made me feel that my instinct of lumping VIDs with Sainsbury was somewhat vindicated Sainsbury of course was from an era where machine learning ML did not yet exist VIDs having been born into an age where ML exists have a far better chance of success than Sainsbury Agostini and Metheringham three famous reachfrequencyduplication heuristics of the last centuryFormer ARF Chief Research Officer Joel Rubinson in the thread wondered if modeling was not inevitable because of the walled gardens and former Chief Research Officer of Nielsen NCS Leslie Wood responded that Nielsen empirically measures many walled gardens by full integration either by direct match or via clean room but 100 empirically in either case The WFAANA would be wise to alter their blueprint to place a higher value on integration of datasets empirically by direct match or clean room allowing the use of modeling only in cases where a particular walled garden is not willing to integrate with any measurerThe modeling which WFAANA ought to recommend should be a method that has demonstrated adequate proof in a sufficient series of diverse cases that the model approximates reality in terms of complex duplication patterns among a host of platforms and target audiences Rubinson points out that while he respects all models the choice of Dirac equations for the GoogleFacebookWFAANA method is not the best choice In the thread he reports that Dirac is specifically known to not preserve internal relationships Rubinson has devised a different mathematical model which he argues will outperform the Diracbased modeling in the cases where modeling is the only option the industry is left with for specific walled gardensMeanwhile it appears thanks to Jim Spaeth and to Michael Vinson that the original formulation from Google did start out assuming quotindependencequot random probability but has already learned that a process for bringing in empirical guardrails is necessary to achieve the degree of accuracy desired So there may be new data on VIDs which we may be able to bring to light in upcoming articles which could demonstrate an acceptable degree of agreement with empirical benchmarksIn this short space Iamp8217ve only lightly touched on the richness of the discussion which arose naturally among some of the finest minds in the business Our industry has no shortage of fine minds It might not be as well represented in the daytoday handtohand as it is in this ARF compilation of the spontaneous email thread You will recognize many famous names in our fieldPosted at MediaVillage through the Thought Leadership selfpublishing platformClick the social buttons to share this story with colleagues and friendsThe opinions expressed here are the author39s views and do not necessarily represent the views of MediaVillageorgMyersBizNet

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Most Experts Agree That Audience Duplication Needs Regular Evidence of Empirical Grounding

Author

Publish date

Channel

Archived in

Bill Harvey

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Most Experts Agree That Audience Duplication Needs Regular Evidence of Empirical Grounding

Enjoying This Commentary? There's More to Love

Bill Harvey

Cookies