Probability Samples Proven More Accurate

The balancing of long term brand strength versus current sales has never been easy and historically the dominant KPI of the 20th Century was brand persuasion measures related to a quasireligious belief that the sales produced by advertising would never be measurable except for direct responseI contributed to the swing to measuring adproduced sales when I cofounded TRA in 2005 and especially in the years that followed in which many companies copied TRAamp8217s use of big data convenience samples to effectively bring multiple regression analysis down from markets as in Marketing Mix Modeling aka amp8220MMMamp8221 to households Digital with its interactivity was all much more readable than oneway media and so thousands of practitioners became able to read sales effects at varying levels of accuracyThe most serious inaccuracy comes from not measuring all stimuli Whatever stimuli you leave out of an analysis amp8211 their effects get credited to the stimuli you are measuring So when a major digital platform says they got you an ROAS of 33 they are taking credit for every other tactic you were also using at the same time you were using their platform One of my clients showed me the results of a campaign as reported by such a platform vs as reported by one of the top MMMbig data matching measurers who had been given all the stimuli data by the client including price promotion all media and all other variables which could have affected sales Where the platform had attributed over 33 ROAS to themselves MMMbig data showed that platform had an ROAS of 021 Simply because of including all stimuliThe marketplace has in the past few years become aware of this to the degree that there is a swing back to MMM but not necessarily away from the big data convenience samples with leading players combining both I will have the honor of presenting much more information on this latest trend at the CIMM Summit on October 1011 2023 and hope you can attend in person will not be available virtuallyProbability Samples More Accurate Than Convenience SamplesBack in the Mad Men era the US was a little different from Europe in terms of emphasis on probability samples Back then many European researchers felt that quota samples could do the job as well as probability samples whereas in America the marketing and media intelligentsia had always believed probability samples to be more accurate However no one was ever able to show me evidence to uphold that preference for probability samplesNow the prestigious Pew Research Center has published an important study comparing probability panels with nonprobability panels consistently finding much closer agreement with government benchmarks in the probability samples than in the nonprobability samples as can be seen from this tableThe large differences in accuracy jump out at you in this graphPew researchers noted that despite sophisticated weighting techniques they could not get the nonprobability results to look more like the probability results This is because in my opinion the number of different human psychological characteristics while not infinite is a very large and unknown number and only a relative handful of these characteristics have been measured at scale to the degree necessary for use as weighting variables At TRA we weighted by 104 characteristics and managed to achieve a high correlation with Nielsen ratings 09533 nevertheless the specific audience projections we made were far off Nielsen numbers causing us to limit ourselves to being a measurer of outcomes and of audience skews to factbased purchaser targets but not claiming to be accurate enough for use as a ratings currencyThis Pew study is a landmark in marketing research It fully confirms the World Federation of Advertisers WFA and Association of National Advertisers ANA in their North Star requirements document for crossplatform audience measurement which requires a probability panel to calibrate big data This requirement was echoed from the main stage of the April 2023 Advertising Research Foundation ARF AUDIENCExSCIENCE conference by ANA Executive Vice President Measurement Jackson Bazley who emphasized the absolute requirement demanded by advertisers for probability sample calibration of big data and auditing by MRC specificallyAll six samples measured by Pew in this study are online panels three based on probability samples and three not based on probability samples If there are still people around the world who didnamp8217t believe they needed probability samples for greater accuracy I would now expect that group to largely change their mindsThe US national Nielsen peoplemeter panel is an area probability sample which is the highest form of probability sample superior to probability samples based on random digit dialing telephone number lists or other device listings The other probability sample approaches are still better than nonprobability samples The accuracy leakage in the nonarea probability samples has to do with their lack of completeness Being a spatial approach tied to the empirical world rather than a black box listing is the virtue of area probability designsThe probability samples used in the Pew study were all addressbased which is one form of area probability sample The addressbased approach in all three cases utilized the US Postal Serviceamp8217s Delivery Sequence File which is estimated by Pew to itself be only 9098 complete The Nielsen gold standard area probability sample involved Nielsen reenumeration of all households within primary sampling points which is presumed to be slightly better than the use of post office filesPew does not report the response rates of the nonprobability samples possibly because users of nonprobability samples rarely keep track of response rates For the three probability samples analyzed in this study Pew reports that their response rates against original predesignated samples were 14 3 and 7 By comparison MRC reports that the Nielsen national US TV panel response rate against original predesignated sample is 30Accuracy is expensive Most of the nations on Earth rich or poor continue to invest billions and trillions of dollars collectively to make weapons more accurate What good is a weapon if it is not accurate What good is marketing if it is based on data that is not accuratePosted at MediaVillage through the Thought Leadership selfpublishing platformClick the social buttons to share this story with colleagues and friendsThe opinions expressed here are the author39s views and do not necessarily represent the views of MediaVillageorgMyersBizNet

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Probability Samples Proven More Accurate

Author

Publish date

Channel

Archived in

Bill Harvey

Breaking Barriers: Nada Bradbury’s AD-ID Strategy for Advertising Effectiveness and Women in Tech - By E.B. Moss

More Videos:

SUBSCRIBE FOR FREE MEDIAVILLAGE CONTENT | BECOME A THOUGHT LEADERSHIP CONTRIBUTOR

Probability Samples Proven More Accurate

Enjoying This Commentary? There's More to Love

Bill Harvey

Breaking Barriers: Nada Bradbury’s AD-ID Strategy for Advertising Effectiveness and Women in Tech - By E.B. Moss

Cookies